SB 53: Artificial intelligence models: large developers.
- Session Year: 2025-2026
- House: Senate
- Latest Version Date: 2025-09-29
Current Status:
Passed
(2025-09-29: Chaptered by Secretary of State. Chapter 138, Statutes of 2025.)
Introduced
In Committee
Awaiting First Chamber Vote
Passed First Chamber
In Committee
Awaiting Second Chamber Vote
Passed Second Chamber
Enacted
(1)Existing law generally regulates artificial intelligence, including by requiring, on or before January 1, 2026, and before each time thereafter, that a generative artificial intelligence system or service, or a substantial modification to a generative artificial intelligence system or service, released on or after January 1, 2022, is made publicly available to Californians for use, the developer of the system or service to post on the developers internet website documentation regarding the data used by the developer to train the generative artificial intelligence system or service, as prescribed.
This bill would enact the Transparency in Frontier Artificial Intelligence Act (TFAIA) that would, among other things related to ensuring the safety of a foundation model, as defined, developed by a frontier developer, require a large frontier developer to write, implement, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developers frontier models and describes how the large frontier developer approaches, among other things, incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework. The TFAIA would also require a large frontier developer to transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk, as defined, resulting from internal use of its frontier models, as specified. The TFAIA would require the Office of Emergency Services to establish a mechanism to be used by a frontier developer or a member of the public to report, as prescribed, a critical safety incident, as defined, and would also require the Office of Emergency Services to establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models, as prescribed.
The TFAIA would exempt from the California Public Records Act a report of a critical safety incident submitted to the Office of Emergency Services, a report of assessments of catastrophic risk from internet use, and a covered employee report made pursuant to the whistleblower protections described below.
The TFAIA would impose a civil penalty for noncompliance with the TFAIA to be enforced by the Attorney General, as prescribed.
(2)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(3)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with foundation models, prohibit a frontier developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents a covered employee, as defined, from disclosing, or retaliates against a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses that the frontier developers activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the frontier developer has violated the TFAIA.
This bill would require a large frontier developer to provide a certain internal process through which a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developers activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer violated the TFAIA. The bill would specify provisions particular to the enforcement of those whistleblower protections and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
This bill would preempt any rule, regulation, code, ordinance, or other law adopted by a city, county, city and county, municipality, or local agency on or after January 1, 2025, specifically related to the regulation of frontier developers with respect to their management of catastrophic risk.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
(1)Existing law generally regulates artificial intelligence, including by requiring, on or before January 1, 2026, and before each time thereafter, that a generative artificial intelligence system or service, or a substantial modification to a generative artificial intelligence system or service, released on or after January 1, 2022, is made publicly available to Californians for use, the developer of the system or service to post on the developers internet website documentation regarding the data used by the developer to train the generative artificial intelligence system or service, as prescribed.
This bill would enact the Transparency in Frontier Artificial Intelligence Act (TFAIA) that would, among other things related to ensuring the safety of a foundation model, as defined, developed by a frontier developer, require a large frontier developer to write, implement, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developers frontier models and describes how the large frontier developer approaches, among other things, incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework. The TFAIA would also require a large frontier developer to transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk, as defined, resulting from internal use of its frontier models, as specified. The TFAIA would require the Office of Emergency Services to establish a mechanism to be used by a frontier developer or a member of the public to report, as prescribed, a critical safety incident, as defined, and would also require the Office of Emergency Services to establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models, as prescribed.
The TFAIA would exempt from the California Public Records Act a report of a critical safety incident submitted to the Office of Emergency Services, a report of assessments of catastrophic risk from internet use, and a covered employee report made pursuant to the whistleblower protections described below.
The TFAIA would impose a civil penalty for noncompliance with the TFAIA to be enforced by the Attorney General, as prescribed.
(2)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(3)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with foundation models, prohibit a frontier developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents a covered employee, as defined, from disclosing, or retaliates against a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses that the frontier developers activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the frontier developer has violated the TFAIA.
This bill would require a large frontier developer to provide a certain internal process through which a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developers activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer violated the TFAIA. The bill would specify provisions particular to the enforcement of those whistleblower protections and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
This bill would preempt any rule, regulation, code, ordinance, or other law adopted by a city, county, city and county, municipality, or local agency on or after January 1, 2025, specifically related to the regulation of frontier developers with respect to their management of catastrophic risk.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
(1)Existing law generally regulates artificial intelligence, including by requiring, on or before January 1, 2026, and before each time thereafter, that a generative artificial intelligence system or service, or a substantial modification to a generative artificial intelligence system or service, released on or after January 1, 2022, is made publicly available to Californians for use, the developer of the system or service to post on the developers internet website documentation regarding the data used by the developer to train the generative artificial intelligence system or service, as prescribed.
This bill would enact the Transparency in Frontier Artificial Intelligence Act (TFAIA) that would, among other things related to ensuring the safety of a foundation model model, as defined, developed by a large frontier developer, require a large frontier developer to write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail, frontier AI framework that applies to the large frontier developers frontier models and describes how the large frontier developer approaches, among other things, the testing procedures that the large developer uses to assess catastrophic risks from its foundation models, as specified. incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework. The TFAIA would define foundation model to mean an artificial intelligence model that is trained on a broad data set, designed for generality of output, and adaptable to a wide range of distinctive tasks. also require a large frontier developer to transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk, as defined, resulting from internal use of its frontier models, as specified. The TFAIA would require the Attorney General Office of Emergency Services to establish a mechanism to be used by a large frontier developer or a member of the public to report, as prescribed, a critical safety incident, as defined, and would authorize the Attorney General to periodically update the definition of foundation model, as specified. defined, and would also require the Office of Emergency Services to establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models, as prescribed.
The TFAIA would exempt from the California Public Records Act a report of a critical safety incident submitted to the Attorney General Office of Emergency Services, a report of assessments of catastrophic risk from internet use, and an a covered employee report made pursuant to the whistleblower protections described below.
The bill TFAIA would impose a civil penalty for noncompliance with the TFAIA to be enforced by the Attorney General, as prescribed.
(2)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(3)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with foundation models, prohibit a large frontier developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee a covered employee, as defined, from disclosing, or retaliates against an a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses that the large frontier developers activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer has violated the TFAIA.
This bill would require a large frontier developer to provide a certain internal process through which an a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developers activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk, risk or that the large frontier developer has made a false or misleading statement about its management of catastrophic risk, as prescribed. violated the TFAIA. The bill would specify provisions particular to the enforcement of those whistleblower protections and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
This bill would preempt any rule, regulation, code, ordinance, or other law adopted by a city, county, city and county, municipality, or local agency on or after January 1, 2025, specifically related to the regulation of frontier developers with respect to their management of catastrophic risk.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
(1)Existing law generally regulates artificial intelligence, including by requiring, on or before January 1, 2026, and before each time thereafter, that a generative artificial intelligence system or service, or a substantial modification to a generative artificial intelligence system or service, released on or after January 1, 2022, is made publicly available to Californians for use, the developer of the system or service to post on the developers internet website documentation regarding the data used by the developer to train the generative artificial intelligence system or service, as prescribed.
This bill would enact the Transparency in Frontier Artificial Intelligence Act (TFAIA) that would, among other things related to ensuring the safety of a foundation model developed by a large developer, require a large developer to write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail, among other things, the testing procedures that the large developer uses to assess catastrophic risks from its foundation models, as specified. The TFAIA would define foundation model to mean an artificial intelligence model that is trained on a broad data set, designed for generality of output, and adaptable to a wide range of distinctive tasks. The TFAIA would require the Attorney General to establish a mechanism to be used by a large developer or a member of the public to report, as prescribed, a critical safety incident, as defined, and would authorize the Attorney General to periodically update the definition of foundation model, as specified.
The TFAIA would require, beginning January 1, 2030, and at least annually thereafter, a large developer to retain an independent third-party auditor to produce a report assessing whether the large developer has substantially complied with its safety and security protocol and any instances of substantial noncompliance during the prior year, as specified, and would require, within 30 days after completing an audit, the auditor to transmit to the Attorney General a high-level summary of the report that fairly presents, in all material respects, the outcome of the audit.The TFAIA would exempt from the California Public Records Act a report of a critical safety incident submitted to the Attorney General, General and an employee report made pursuant to the whistleblower protections described below, and a summary of an auditors report. below.
The bill would impose a civil penalty for noncompliance with the TFAIA to be enforced by the Attorney General, as prescribed.
(2)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(3)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with foundation models, prohibit a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the large developers activities pose a catastrophic risk or that the large developer has violated the TFAIA.
This bill would require a large developer to provide a certain internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developers activities present a catastrophic risk, or that the developer has made a false or misleading statement about its management of catastrophic risk, as prescribed. The bill would specify provisions particular to the enforcement of those whistleblower protections and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
(1)Existing law generally regulates artificial intelligence, including by requiring, on or before January 1, 2026, and before each time thereafter, that a generative artificial intelligence system or service, or a substantial modification to a generative artificial intelligence system or service, released on or after January 1, 2022, is made publicly available to Californians for use, the developer of the system or service to post on the developers internet website documentation regarding the data used by the developer to train the generative artificial intelligence system or service, as prescribed.
This bill would enact the Transparency in Frontier Artificial Intelligence Act (TFAIA) that would, among other things related to ensuring the safety of a foundation model developed by a large developer, require a large developer to write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail, among other things, the testing procedures that the large developer uses to assess catastrophic risks from its foundation models, as specified. The TFAIA would define foundation model to mean an artificial intelligence model that is trained on a broad data set, designed for generality of output, and adaptable to a wide range of distinctive tasks. The TFAIA would require the Attorney General to establish a mechanism to be used by a large developer or a member of the public to report, as prescribed, a critical safety incident, as defined, and would authorize the Attorney General to periodically update the definition of foundation model, as specified. The bill would impose a civil penalty for noncompliance to be enforced by the Attorney General.
The TFAIA would require, beginning January 1, 2030, and at least annually thereafter, a large developer to retain an independent third-party auditor to produce a report assessing whether the large developer has substantially complied with its safety and security protocol and any instances of substantial noncompliance during the prior year, as specified, and would require, within 30 days after completing an audit, the auditor to transmit to the Attorney General a high-level summary of the report that fairly presents, in all material respects, the outcome of the audit.
The TFAIA would exempt from the California Public Records Act a report of a critical safety incident submitted to the Attorney General, an employee report made pursuant to the whistleblower protections described below, and a summary of an auditors report.
The bill would impose a civil penalty for noncompliance with the TFAIA to be enforced by the Attorney General, as prescribed.
(2)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(3)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with foundation models, prohibit a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the large developers activities pose a catastrophic risk or that the large developer has violated the TFAIA.
This bill would require a large developer to provide a certain internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developers activities present a catastrophic risk, or that the developer has made a false or misleading statement about its management of catastrophic risk, as prescribed. The bill would specify provisions particular to the enforcement of those whistleblower protections and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
(1)Existing law generally regulates artificial intelligence, including by requiring, on or before January 1, 2026, and before each time thereafter, that a generative artificial intelligence system or service, or a substantial modification to a generative artificial intelligence system or service, released on or after January 1, 2022, is made publicly available to Californians for use, the developer of the system or service to post on the developers internet website documentation regarding the data used by the developer to train the generative artificial intelligence system or service, as prescribed.
This bill would enact the Transparency in Frontier Artificial Intelligence Act (TFAIA) that would, among other things related to ensuring the safety of a foundation model developed by a large developer, require a large developer to write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail, among other things, the testing procedures that the large developer uses to assess catastrophic risks from its foundation models, as specified. The TFAIA would define foundation model to mean an artificial intelligence model that is trained on a broad data set, designed for generality of output, and adaptable to a wide range of distinctive tasks. The TFAIA would require the Attorney General to establish a mechanism to be used by a large developer or a member of the public to report, as prescribed, a critical safety incident, as defined, and would authorize the Attorney General to periodically update the definition of foundation model, as specified. The bill would impose a civil penalty for noncompliance to be enforced by the Attorney General.
(1)(2)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework, framework and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(2)(3)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with certain artificial intelligence foundation models, prohibit a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the large developers activities pose a critical catastrophic risk or that the large developer has made false or misleading statements about its management of critical risk. The violated the TFAIA.
This bill would define critical risk to mean a foreseeable and material risk that a developers development, storage, or deployment of a foundation model, as defined, will result in the death of, or serious injury to, more than 100 people, or more than $1 billion in damage to rights in money or property, as provided. The bill would also require a large developer to provide a certain internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developers activities present a critical catastrophic risk, or that the developer has made a false or misleading statement about its management of critical catastrophic risk, as prescribed. The bill would specify provisions particular to the enforcement of those whistleblower protections and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
(1)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework, and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(2)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with certain artificial intelligence models, prohibit a developer from making, adopting, or enforcing enforcing, or entering into a rule, regulation, or policy policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the developers activities pose a critical risk or that the developer has made false or misleading statements about its management of critical risk. The bill would define critical risk to mean a foreseeable and material risk that a developers development, storage, or deployment of a foundation model, as defined, will result in the death of, or serious injury to, more than 100 people, or more than $1 billion in damage to rights in money or property, as provided. The bill would also require a developer to provide a certain internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developers activities present a critical risk, or that the developer has made a false or misleading statement about its management of critical risk, as prescribed. The bill would specify provisions particular to enforcement and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
(1)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework. framework, and would dissolve the consortium upon submission of that report. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(2)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with certain artificial intelligence models, prohibit a developer from making, adopting, or enforcing a rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the developers activities pose a critical risk or that the developer has made false or misleading statements about its management of critical risk. The bill would define critical risk to mean a foreseeable and material risk that a developers development, storage, or deployment of a foundation model, as defined, will result in the death of, or serious injury to, more than 100 people, or more than $1 billion in damage to rights in money or property, as provided. The bill would also require a developer to provide a certain internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developers activities present a critical risk, as prescribed. The bill would specify provisions particular to enforcement and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
(1)Existing law establishes the Department of Technology within the Government Operations Agency. Existing law requires the department to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would establish within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would require the Government Operations Agency to, on or before January 1, 2027, submit a report from the consortium to the Legislature with that framework. The bill would make those provisions operative only upon an appropriation in a budget act, or other measure, for its purposes.
(2)Existing law prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill would, among other things related to protecting whistleblowers working with certain artificial intelligence models, prohibit a developer from making, adopting, or enforcing a rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the developers activities pose a critical risk or that the developer has made false or misleading statements about its management of critical risk. The bill would define critical risk to mean a foreseeable and material risk that a developers development, storage, or deployment of a foundation model, as defined, will result in the death of, or serious injury to, more than 100 people, or more than $1 billion in damage to rights in money or property, as provided. The bill would also require a developer to provide a certain internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developers activities present a critical risk, as prescribed. The bill would specify provisions particular to enforcement and would authorize attorneys fees to a plaintiff who brings a successful action for a violation.
Existing law requires the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.Existing law defines automated decision system as a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence (AI) that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. Existing law defines artificial intelligence as an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.Existing law, the Generative Artificial Intelligence Accountability Act, among other things, requires the Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the Department of Human Resources, to update the report to the Governor, as required by Executive Order No. N-12-23, as prescribed, and requires the Office of Emergency Services to perform, as appropriate, a risk analysis of potential threats posed by the use of generative AI to Californias critical infrastructure, including those that could lead to mass casualty events.This bill would declare the intent of the Legislature to enact legislation that would establish safeguards for the development of AI frontier models and that would build state capacity for the use of AI, that may include, but is not limited to, the findings of the Joint California Policy Working Group on AI Frontier Models established by the Governor.Existing law requires the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
Existing law defines automated decision system as a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence (AI) that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. Existing law defines artificial intelligence as an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
Existing law, the Generative Artificial Intelligence Accountability Act, among other things, requires the Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the Department of Human Resources, to update the report to the Governor, as required by Executive Order No. N-12-23, as prescribed, and requires the Office of Emergency Services to perform, as appropriate, a risk analysis of potential threats posed by the use of generative AI to Californias critical infrastructure, including those that could lead to mass casualty events.
This bill would declare the intent of the Legislature to enact legislation that would establish safeguards for the development of AI frontier models and that would build state capacity for the use of AI, that may include, but is not limited to, the findings of the Joint California Policy Working Group on AI Frontier Models established by the Governor.
The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research through companies large and small and through the states remarkable public and private universities.
(b)Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d)Targeted interventions to support effective artificial intelligence governance should balance the technologys benefits and the potential for material risks.
(e)In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(f)As industry actors conduct internal research on their technologies impacts, public trust in these technologies would significantly benefit from access to information regarding, and increased awareness of, frontier AI capabilities.
(g)Greater transparency can also advance accountability, competition, and public trust.
(h)Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(i)Incident reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(j)Unless they are developed with careful diligence and reasonable precaution, there is concern that advanced artificial intelligence systems could have capabilities that pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(k)With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to serious risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(l)While the major artificial intelligence developers have already voluntarily established the creation, use, and publication of frontier AI frameworks as an industry best practice, not all developers are providing reporting that is consistent and sufficient to ensure necessary transparency and protection of the public. Mandatory, standardized, and objective reporting by frontier developers is required to provide the government and the public with timely and accurate information.
(m)Timely reporting of critical safety incidents to the government is essential to ensure that public authorities are promptly informed of ongoing and emerging risks to public safety. This reporting enables the government to monitor, assess, and respond effectively in the event that advanced capabilities emerge in frontier artificial intelligence models that may pose a threat to the public.
(n)In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(o)The recent release of the Governors California Report on Frontier AI Policy and testimony from legislative hearings on artificial intelligence before the Legislature reflect the advances in AI model capabilities that could pose potential catastrophic risk in frontier artificial intelligence, which this act aims to address.
(p)It is the intent of the Legislature to create more transparency, but collective safety will depend in part on frontier developers taking due care in their development and deployment of frontier models proportional to the scale of the foreseeable risks.
SEC. 2.Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:25.1.Transparency in Frontier Artificial Intelligence Act22757.10.This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11.For purposes of this chapter:
(a)Affiliate means a person controlling, controlled by, or under common control with a specified person, directly or indirectly, through one or more intermediaries.
(b)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)(1)Catastrophic risk means a foreseeable and material risk that a frontier developers development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident involving a frontier model doing any of the following:
(A)Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(B)Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if the conduct had been committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(C)Evading the control of its frontier developer or user.
(2)Catastrophic risk does not include a foreseeable and material risk from any of the following:
(A)Information that a frontier model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
(B)Lawful activity of the federal government.
(C)Harm caused by a frontier model in combination with other software if the frontier model did not materially contribute to the harm.
(d)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model that results in death or bodily injury.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a frontier model causing death or bodily injury.
(4)A frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.
(e)(1)Deploy means to make a frontier model available to a third party for use, modification, copying, or combination with other software.
(2)Deploy does not include making a frontier model available to a third party for the primary purpose of developing or evaluating the frontier model.
(f)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data set.
(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(g)Frontier AI framework means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
(h)Frontier developer means a person who has trained, or initiated the training of, a frontier model, with respect to which the person has used, or intends to use, at least as much computing power to train the frontier model as would meet the technical specifications found in subdivision (i).
(i)(1)Frontier model means a foundation model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
(2)The quantity of computing power described in paragraph (1) shall include computing for the original training run and for any subsequent fine-tuning, reinforcement learning, or other material modifications the developer applies to a preceding foundation model.
(j)Large frontier developer means a frontier developer that together with its affiliates collectively had annual gross revenues in excess of five hundred million dollars ($500,000,000) in the preceding calendar year.
(k)Model weight means a numerical parameter in a frontier model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(l)Property means tangible or intangible property.
22757.12.(a)A large frontier developer shall write, implement, comply with, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developers frontier models and describes how the large frontier developer approaches all of the following:
(1)Incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework.
(2)Defining and assessing thresholds used by the large frontier developer to identify and assess whether a frontier model has capabilities that could pose a catastrophic risk, which may include multiple-tiered thresholds.
(3)Applying mitigations to address the potential for catastrophic risks based on the results of assessments undertaken pursuant to paragraph (2).
(4)Reviewing assessments and adequacy of mitigations as part of the decision to deploy a frontier model or use it extensively internally.
(5)Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks.
(6)Revisiting and updating the frontier AI framework, including any criteria that trigger updates and how the large frontier developer determines when its frontier models are substantially modified enough to require disclosures pursuant to subdivision (c).
(7)Cybersecurity practices to secure unreleased model weights from unauthorized modification or transfer by internal or external parties.
(8)Identifying and responding to critical safety incidents.
(9)Instituting internal governance practices to ensure implementation of these processes.
(10)Assessing and managing catastrophic risk resulting from the internal use of its frontier models, including risks resulting from a frontier model circumventing oversight mechanisms.
(b)(1)A large frontier developer shall review and, as appropriate, update its frontier AI framework at least once per year.
(2)If a large frontier developer makes a material modification to its frontier AI framework, the large frontier developer shall clearly and conspicuously publish the modified frontier AI framework and a justification for that modification within 30 days.
(c)(1)Before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, a frontier developer shall clearly and conspicuously publish on its internet website a transparency report containing all of the following:
(A)The internet website of the frontier developer.
(B)A mechanism that enables a natural person to communicate with the frontier developer.
(C)The release date of the frontier model.
(D)The languages supported by the frontier model.
(E)The modalities of output supported by the frontier model.
(F)The intended uses of the frontier model.
(G)Any generally applicable restrictions or conditions on uses of the frontier model.
(2)Before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, a large frontier developer shall include in the transparency report required by paragraph (1) summaries of all of the following:
(A)Assessments of catastrophic risks from the frontier model conducted pursuant to the large frontier developers frontier AI framework.
(B)The results of those assessments.
(C)The extent to which third-party evaluators were involved.
(D)Other steps taken to fulfill the requirements of the frontier AI framework with respect to the frontier model.
(3)A frontier developer that publishes the information described in paragraph (1) or (2) as part of a larger document, including a system card or model card, shall be deemed in compliance with the applicable paragraph.
(4)A frontier developer is encouraged, but not required, to make disclosures described in this subdivision that are consistent with, or superior to, industry best practices.
(d)A large frontier developer shall transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months or pursuant to another reasonable schedule specified by the large frontier developer and communicated in writing to the Office of Emergency Services with written updates, as appropriate.
(e)(1)(A)A frontier developer shall not make a materially false or misleading statement about catastrophic risk from its frontier models or its management of catastrophic risk.
(B)A large frontier developer shall not make a materially false or misleading statement about its implementation of, or compliance with, its frontier AI framework.
(2)This subdivision does not apply to a statement that was made in good faith and was reasonable under the circumstances.
(f)(1)When a frontier developer publishes documents to comply with this section, the frontier developer may make redactions to those documents that are necessary to protect the frontier developers trade secrets, the frontier developers cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2)If a frontier developer redacts information in a document pursuant to this subdivision, the frontier developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13.(a)The Office of Emergency Services shall establish a mechanism to be used by a frontier developer or a member of the public to report a critical safety incident that includes all of the following:
(1)The date of the critical safety incident.
(2)The reasons the incident qualifies as a critical safety incident.
(3)A short and plain statement describing the critical safety incident.
(4)Whether the incident was associated with internal use of a frontier model.
(b)(1)The Office of Emergency Services shall establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models.
(2)The Office of Emergency Services shall take all necessary precautions to limit access to any reports related to internal use of frontier models to only personnel with a specific need to know the information and to protect the reports from unauthorized access.
(c)(1)Subject to paragraph (2), a frontier developer shall report any critical safety incident pertaining to one or more of its frontier models to the Office of Emergency Services within 15 days of discovering the critical safety incident.
(2)If a frontier developer discovers that a critical safety incident poses an imminent risk of death or serious physical injury, the frontier developer shall disclose that incident within 24 hours to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and as required by law.
(3)A frontier developer that discovers information about a critical safety incident after filing the initial report required by this subdivision may file an amended report.
(4)A frontier developer is encouraged, but not required, to report critical safety incidents pertaining to foundation models that are not frontier models.
(d)The Office of Emergency Services shall review critical safety incident reports submitted by frontier developers and may review reports submitted by members of the public.
(e)(1)The Attorney General or the Office of Emergency Services may transmit reports of critical safety incidents and reports from covered employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code to the Legislature, the Governor, the federal government, or appropriate state agencies.
(2)The Attorney General or the Office of Emergency Services shall strongly consider any risks related to trade secrets, public safety, cybersecurity of a frontier developer, or national security when transmitting reports.
(f)A report of a critical safety incident submitted to the Office of Emergency Services pursuant to this section, a report of assessments of catastrophic risk from internal use pursuant to Section 22757.12, and a covered employee report made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(g)(1)Beginning January 1, 2027, and annually thereafter, the Office of Emergency Services shall produce a report with anonymized and aggregated information about critical safety incidents that have been reviewed by the Office of Emergency Services since the preceding report.
(2)The Office of Emergency Services shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a frontier developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Office of Emergency Services shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795, and to the Governor.
(h)The Office of Emergency Services may adopt regulations designating one or more federal laws, regulations, or guidance documents that meet all of the following conditions for the purposes of subdivision (i):
(1)(A)The law, regulation, or guidance document imposes or states standards or requirements for critical safety incident reporting that are substantially equivalent to, or stricter than, those required by this section.
(B)The law, regulation, or guidance document described in subparagraph (A) does not need to require critical safety incident reporting to the State of California.
(2)The law, regulation, or guidance document is intended to assess, detect, or mitigate the catastrophic risk.
(i)(1)A frontier developer that intends to comply with this section by complying with the requirements of, or meeting the standards stated by, a federal law, regulation, or guidance document designated pursuant to subdivision (h) shall declare its intent to do so to the Office of Emergency Services.
(2)After a frontier developer has declared its intent pursuant to paragraph (1), both of the following apply:
(A)The frontier developer shall be deemed in compliance with this section to the extent that the frontier developer meets the standards of, or complies with the requirements imposed or stated by, the designated federal law, regulation, or guidance document until the frontier developer declares the revocation of that intent to the Office of Emergency Services or the Office of Emergency Services revokes a relevant regulation pursuant to subdivision (j).
(B)The failure by a frontier developer to meet the standards of, or comply with the requirements stated by, the federal law, regulation, or guidance document designated pursuant to subdivision (h) shall constitute a violation of this chapter.
(j)The Office of Emergency Services shall revoke a regulation adopted under subdivision (h) if the requirements of subdivision (h) are no longer met.
22757.14.(a)On or before January 1, 2027, and annually thereafter, the Department of Technology shall assess recent evidence and developments relevant to the purposes of this chapter and shall make recommendations about whether and how to update any of the following definitions for the purposes of this chapter to ensure that they accurately reflect technological developments, scientific literature, and widely accepted national and international standards:
(1)Frontier model so that it applies to foundation models at the frontier of artificial intelligence development.
(2)Frontier developer so that it applies to developers of frontier models who are themselves at the frontier of artificial intelligence development.
(3)Large frontier developer so that it applies to well-resourced frontier developers.
(b)In making recommendations pursuant to this section, the Department of Technology shall take into account all of the following:
(1)Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk and shall align with a definition adopted in a federal law or regulation to the extent that it is consistent with the purposes of this chapter.
(2)Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3)The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the definition as a frontier developer or as a large frontier developer with an aim toward allowing earlier determinations if possible.
(4)The complexity of determining whether a person or foundation model is covered, with an aim toward allowing simpler determinations if possible.
(5)The external verifiability of determining whether a person or foundation model is covered, with an aim toward definitions that are verifiable by parties other than the frontier developer.
(c)Upon developing recommendations pursuant to this section, the Department of Technology shall submit a report to the Legislature, pursuant to Section 9795 of the Government Code, with those recommendations.
(d)(1)Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about reports from covered employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code that have been reviewed by the Attorney General since the preceding report.
(2)The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a frontier developer, confidentiality of a covered employee, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Attorney General shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795 of the Government Code, and to the Governor.
22757.15.(a)A large frontier developer that fails to publish or transmit a compliant document required to be published or transmitted under this chapter, makes a statement in violation of subdivision (e) of Section 22757.12, fails to report an incident as required by Section 22757.13, or fails to comply with its own frontier AI framework shall be subject to a civil penalty in an amount dependent upon the severity of the violation that does not exceed one million dollars ($1,000,000) per violation.
(b)A civil penalty described in this section shall be recovered in a civil action brought only by the Attorney General.
22757.16.The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
SEC. 3.Section 11546.8 is added to the Government Code, to read:11546.8.(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1)Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2)Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3)Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4)Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Catastrophic Risks in AI Foundation Models1107.For purposes of this chapter:
(a)(1)Catastrophic risk means a foreseeable and material risk that a frontier developers development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident involving a foundation model doing any of the following:
(A)Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(B)Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(C)Evading the control of its frontier developer or user.
(2)Catastrophic risk does not include a foreseeable and material risk from any of the following:
(A)Information that a foundation model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
(B)Lawful activity of the federal government.
(C)Harm caused by a foundation model in combination with other software where the foundation model did not materially contribute to the harm.
(b)Covered employee means an employee responsible for assessing, managing, or addressing risk of critical safety incidents.
(c)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of the model weights of a foundation model that results in death, bodily injury, or damage to, or loss of, property.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation model causing death or bodily injury.
(4)A foundation model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.
(d)Foundation model has the meaning defined in Section 22757.11 of the Business and Professions Code.
(e)Frontier developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(f)Large frontier developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
1107.1.(a)A frontier developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents a covered employee from disclosing, or retaliates against a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses either of the following:
(1)The frontier developers activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk.
(2)The frontier developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b)A frontier developer shall not enter into a contract that prevents a covered employee from making a disclosure protected under Section 1102.5.
(c)A covered employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(d)A frontier developer shall provide a clear notice to all covered employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the frontier developer a notice to all covered employees of their rights under this section, ensuring that any new covered employee receives equivalent notice, and ensuring that any covered employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each covered employee of the covered employees rights under this section and ensuring that the notice is received and acknowledged by all of those covered employees.
(e)(1)A large frontier developer shall provide a reasonable internal process through which a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developers activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large frontier developers investigation of the disclosure and the actions taken by the large frontier developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large frontier developer at least once each quarter.
(B)If a covered employee has alleged wrongdoing by an officer or director of the large frontier developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(f)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(g)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the covered employee, the frontier developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the covered employee had not engaged in activities protected by this section.
(h)(1)In a civil action or administrative proceeding brought pursuant to this section, a covered employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other covered employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a frontier developer from disciplining or terminating a covered employee for conduct that is unrelated to the claim of the retaliation.
(i)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(j)(1)This section does not impair or limit the applicability of Section 1102.5, including with respect to the rights of employees who are not covered employees to report violations of this chapter or Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
1107.2.The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
SEC. 5.(a)The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b)This act shall be liberally construed to effectuate its purposes.
(c)The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d)This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a frontier developer.
(e)This act shall not apply to the extent that it is preempted by federal law.
(f)This act preempts any rule, regulation, code, ordinance, or other law adopted by a city, county, city and county, municipality, or local agency on or after January 1, 2025, specifically related to the regulation of frontier developers with respect to their management of catastrophic risk.
SEC. 6.The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports, assessments of risks from internal use, and reports from covered employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research through companies large and small and through the states remarkable public and private universities.
(b)Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d)Targeted interventions to support effective artificial intelligence governance should balance the technologys benefits and the potential for material risks.
(e)In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(f)As industry actors conduct internal research on their technologies impacts, public trust in these technologies would significantly benefit from access to information regarding, and increased awareness of, frontier AI capabilities.
(g)Greater transparency can also advance accountability, competition, and public trust.
(h)Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(i)Incident reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(j)Unless they are developed with careful diligence and reasonable precaution, there is concern that advanced artificial intelligence systems could have capabilities that pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(k)With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to serious risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(l)While the major artificial intelligence developers have already voluntarily established the creation, use, and publication of frontier AI frameworks as an industry best practice, not all developers are providing reporting that is consistent and sufficient to ensure necessary transparency and protection of the public. Mandatory, standardized, and objective reporting by frontier developers is required to provide the government and the public with timely and accurate information.
(m)Timely reporting of critical safety incidents to the government is essential to ensure that public authorities are promptly informed of ongoing and emerging risks to public safety. This reporting enables the government to monitor, assess, and respond effectively in the event that advanced capabilities emerge in frontier artificial intelligence models that may pose a threat to the public.
(n)In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(o)The recent release of the Governors California Report on Frontier AI Policy and testimony from legislative hearings on artificial intelligence before the Legislature reflect the advances in AI model capabilities that could pose potential catastrophic risk in frontier artificial intelligence, which this act aims to address.
(p)It is the intent of the Legislature to create more transparency, but collective safety will depend in part on frontier developers taking due care in their development and deployment of frontier models proportional to the scale of the foreseeable risks.
SEC. 2.Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:25.1.Transparency in Frontier Artificial Intelligence Act22757.10.This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11.For purposes of this chapter:
(a)Affiliate means a person controlling, controlled by, or under common control with a specified person, directly or indirectly, through one or more intermediaries.
(b)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)(1)Catastrophic risk means a foreseeable and material risk that a frontier developers development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident involving a frontier model doing any of the following:
(A)Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(B)Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if the conduct had been committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(C)Evading the control of its frontier developer or user.
(2)Catastrophic risk does not include a foreseeable and material risk from any of the following:
(A)Information that a frontier model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
(B)Lawful activity of the federal government.
(C)Harm caused by a frontier model in combination with other software if the frontier model did not materially contribute to the harm.
(d)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model that results in death or bodily injury.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a frontier model causing death or bodily injury.
(4)A frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.
(e)(1)Deploy means to make a frontier model available to a third party for use, modification, copying, or combination with other software.
(2)Deploy does not include making a frontier model available to a third party for the primary purpose of developing or evaluating the frontier model.
(f)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data set.
(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(g)Frontier AI framework means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
(h)Frontier developer means a person who has trained, or initiated the training of, a frontier model, with respect to which the person has used, or intends to use, at least as much computing power to train the frontier model as would meet the technical specifications found in subdivision (i).
(i)(1)Frontier model means a foundation model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
(2)The quantity of computing power described in paragraph (1) shall include computing for the original training run and for any subsequent fine-tuning, reinforcement learning, or other material modifications the developer applies to a preceding foundation model.
(j)Large frontier developer means a frontier developer that together with its affiliates collectively had annual gross revenues in excess of five hundred million dollars ($500,000,000) in the preceding calendar year.
(k)Model weight means a numerical parameter in a frontier model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(l)Property means tangible or intangible property.
22757.12.(a)A large frontier developer shall write, implement, comply with, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developers frontier models and describes how the large frontier developer approaches all of the following:
(1)Incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework.
(2)Defining and assessing thresholds used by the large frontier developer to identify and assess whether a frontier model has capabilities that could pose a catastrophic risk, which may include multiple-tiered thresholds.
(3)Applying mitigations to address the potential for catastrophic risks based on the results of assessments undertaken pursuant to paragraph (2).
(4)Reviewing assessments and adequacy of mitigations as part of the decision to deploy a frontier model or use it extensively internally.
(5)Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks.
(6)Revisiting and updating the frontier AI framework, including any criteria that trigger updates and how the large frontier developer determines when its frontier models are substantially modified enough to require disclosures pursuant to subdivision (c).
(7)Cybersecurity practices to secure unreleased model weights from unauthorized modification or transfer by internal or external parties.
(8)Identifying and responding to critical safety incidents.
(9)Instituting internal governance practices to ensure implementation of these processes.
(10)Assessing and managing catastrophic risk resulting from the internal use of its frontier models, including risks resulting from a frontier model circumventing oversight mechanisms.
(b)(1)A large frontier developer shall review and, as appropriate, update its frontier AI framework at least once per year.
(2)If a large frontier developer makes a material modification to its frontier AI framework, the large frontier developer shall clearly and conspicuously publish the modified frontier AI framework and a justification for that modification within 30 days.
(c)(1)Before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, a frontier developer shall clearly and conspicuously publish on its internet website a transparency report containing all of the following:
(A)The internet website of the frontier developer.
(B)A mechanism that enables a natural person to communicate with the frontier developer.
(C)The release date of the frontier model.
(D)The languages supported by the frontier model.
(E)The modalities of output supported by the frontier model.
(F)The intended uses of the frontier model.
(G)Any generally applicable restrictions or conditions on uses of the frontier model.
(2)Before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, a large frontier developer shall include in the transparency report required by paragraph (1) summaries of all of the following:
(A)Assessments of catastrophic risks from the frontier model conducted pursuant to the large frontier developers frontier AI framework.
(B)The results of those assessments.
(C)The extent to which third-party evaluators were involved.
(D)Other steps taken to fulfill the requirements of the frontier AI framework with respect to the frontier model.
(3)A frontier developer that publishes the information described in paragraph (1) or (2) as part of a larger document, including a system card or model card, shall be deemed in compliance with the applicable paragraph.
(4)A frontier developer is encouraged, but not required, to make disclosures described in this subdivision that are consistent with, or superior to, industry best practices.
(d)A large frontier developer shall transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months or pursuant to another reasonable schedule specified by the large frontier developer and communicated in writing to the Office of Emergency Services with written updates, as appropriate.
(e)(1)(A)A frontier developer shall not make a materially false or misleading statement about catastrophic risk from its frontier models or its management of catastrophic risk.
(B)A large frontier developer shall not make a materially false or misleading statement about its implementation of, or compliance with, its frontier AI framework.
(2)This subdivision does not apply to a statement that was made in good faith and was reasonable under the circumstances.
(f)(1)When a frontier developer publishes documents to comply with this section, the frontier developer may make redactions to those documents that are necessary to protect the frontier developers trade secrets, the frontier developers cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2)If a frontier developer redacts information in a document pursuant to this subdivision, the frontier developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13.(a)The Office of Emergency Services shall establish a mechanism to be used by a frontier developer or a member of the public to report a critical safety incident that includes all of the following:
(1)The date of the critical safety incident.
(2)The reasons the incident qualifies as a critical safety incident.
(3)A short and plain statement describing the critical safety incident.
(4)Whether the incident was associated with internal use of a frontier model.
(b)(1)The Office of Emergency Services shall establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models.
(2)The Office of Emergency Services shall take all necessary precautions to limit access to any reports related to internal use of frontier models to only personnel with a specific need to know the information and to protect the reports from unauthorized access.
(c)(1)Subject to paragraph (2), a frontier developer shall report any critical safety incident pertaining to one or more of its frontier models to the Office of Emergency Services within 15 days of discovering the critical safety incident.
(2)If a frontier developer discovers that a critical safety incident poses an imminent risk of death or serious physical injury, the frontier developer shall disclose that incident within 24 hours to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and as required by law.
(3)A frontier developer that discovers information about a critical safety incident after filing the initial report required by this subdivision may file an amended report.
(4)A frontier developer is encouraged, but not required, to report critical safety incidents pertaining to foundation models that are not frontier models.
(d)The Office of Emergency Services shall review critical safety incident reports submitted by frontier developers and may review reports submitted by members of the public.
(e)(1)The Attorney General or the Office of Emergency Services may transmit reports of critical safety incidents and reports from covered employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code to the Legislature, the Governor, the federal government, or appropriate state agencies.
(2)The Attorney General or the Office of Emergency Services shall strongly consider any risks related to trade secrets, public safety, cybersecurity of a frontier developer, or national security when transmitting reports.
(f)A report of a critical safety incident submitted to the Office of Emergency Services pursuant to this section, a report of assessments of catastrophic risk from internal use pursuant to Section 22757.12, and a covered employee report made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(g)(1)Beginning January 1, 2027, and annually thereafter, the Office of Emergency Services shall produce a report with anonymized and aggregated information about critical safety incidents that have been reviewed by the Office of Emergency Services since the preceding report.
(2)The Office of Emergency Services shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a frontier developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Office of Emergency Services shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795, and to the Governor.
(h)The Office of Emergency Services may adopt regulations designating one or more federal laws, regulations, or guidance documents that meet all of the following conditions for the purposes of subdivision (i):
(1)(A)The law, regulation, or guidance document imposes or states standards or requirements for critical safety incident reporting that are substantially equivalent to, or stricter than, those required by this section.
(B)The law, regulation, or guidance document described in subparagraph (A) does not need to require critical safety incident reporting to the State of California.
(2)The law, regulation, or guidance document is intended to assess, detect, or mitigate the catastrophic risk.
(i)(1)A frontier developer that intends to comply with this section by complying with the requirements of, or meeting the standards stated by, a federal law, regulation, or guidance document designated pursuant to subdivision (h) shall declare its intent to do so to the Office of Emergency Services.
(2)After a frontier developer has declared its intent pursuant to paragraph (1), both of the following apply:
(A)The frontier developer shall be deemed in compliance with this section to the extent that the frontier developer meets the standards of, or complies with the requirements imposed or stated by, the designated federal law, regulation, or guidance document until the frontier developer declares the revocation of that intent to the Office of Emergency Services or the Office of Emergency Services revokes a relevant regulation pursuant to subdivision (j).
(B)The failure by a frontier developer to meet the standards of, or comply with the requirements stated by, the federal law, regulation, or guidance document designated pursuant to subdivision (h) shall constitute a violation of this chapter.
(j)The Office of Emergency Services shall revoke a regulation adopted under subdivision (h) if the requirements of subdivision (h) are no longer met.
22757.14.(a)On or before January 1, 2027, and annually thereafter, the Department of Technology shall assess recent evidence and developments relevant to the purposes of this chapter and shall make recommendations about whether and how to update any of the following definitions for the purposes of this chapter to ensure that they accurately reflect technological developments, scientific literature, and widely accepted national and international standards:
(1)Frontier model so that it applies to foundation models at the frontier of artificial intelligence development.
(2)Frontier developer so that it applies to developers of frontier models who are themselves at the frontier of artificial intelligence development.
(3)Large frontier developer so that it applies to well-resourced frontier developers.
(b)In making recommendations pursuant to this section, the Department of Technology shall take into account all of the following:
(1)Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk and shall align with a definition adopted in a federal law or regulation to the extent that it is consistent with the purposes of this chapter.
(2)Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3)The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the definition as a frontier developer or as a large frontier developer with an aim toward allowing earlier determinations if possible.
(4)The complexity of determining whether a person or foundation model is covered, with an aim toward allowing simpler determinations if possible.
(5)The external verifiability of determining whether a person or foundation model is covered, with an aim toward definitions that are verifiable by parties other than the frontier developer.
(c)Upon developing recommendations pursuant to this section, the Department of Technology shall submit a report to the Legislature, pursuant to Section 9795 of the Government Code, with those recommendations.
(d)(1)Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about reports from covered employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code that have been reviewed by the Attorney General since the preceding report.
(2)The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a frontier developer, confidentiality of a covered employee, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Attorney General shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795 of the Government Code, and to the Governor.
22757.15.(a)A large frontier developer that fails to publish or transmit a compliant document required to be published or transmitted under this chapter, makes a statement in violation of subdivision (e) of Section 22757.12, fails to report an incident as required by Section 22757.13, or fails to comply with its own frontier AI framework shall be subject to a civil penalty in an amount dependent upon the severity of the violation that does not exceed one million dollars ($1,000,000) per violation.
(b)A civil penalty described in this section shall be recovered in a civil action brought only by the Attorney General.
22757.16.The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
SEC. 3.Section 11546.8 is added to the Government Code, to read:11546.8.(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1)Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2)Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3)Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4)Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Catastrophic Risks in AI Foundation Models1107.For purposes of this chapter:
(a)(1)Catastrophic risk means a foreseeable and material risk that a frontier developers development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident involving a foundation model doing any of the following:
(A)Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(B)Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(C)Evading the control of its frontier developer or user.
(2)Catastrophic risk does not include a foreseeable and material risk from any of the following:
(A)Information that a foundation model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
(B)Lawful activity of the federal government.
(C)Harm caused by a foundation model in combination with other software where the foundation model did not materially contribute to the harm.
(b)Covered employee means an employee responsible for assessing, managing, or addressing risk of critical safety incidents.
(c)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of the model weights of a foundation model that results in death, bodily injury, or damage to, or loss of, property.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation model causing death or bodily injury.
(4)A foundation model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.
(d)Foundation model has the meaning defined in Section 22757.11 of the Business and Professions Code.
(e)Frontier developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(f)Large frontier developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
1107.1.(a)A frontier developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents a covered employee from disclosing, or retaliates against a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses either of the following:
(1)The frontier developers activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk.
(2)The frontier developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b)A frontier developer shall not enter into a contract that prevents a covered employee from making a disclosure protected under Section 1102.5.
(c)A covered employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(d)A frontier developer shall provide a clear notice to all covered employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the frontier developer a notice to all covered employees of their rights under this section, ensuring that any new covered employee receives equivalent notice, and ensuring that any covered employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each covered employee of the covered employees rights under this section and ensuring that the notice is received and acknowledged by all of those covered employees.
(e)(1)A large frontier developer shall provide a reasonable internal process through which a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developers activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large frontier developers investigation of the disclosure and the actions taken by the large frontier developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large frontier developer at least once each quarter.
(B)If a covered employee has alleged wrongdoing by an officer or director of the large frontier developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(f)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(g)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the covered employee, the frontier developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the covered employee had not engaged in activities protected by this section.
(h)(1)In a civil action or administrative proceeding brought pursuant to this section, a covered employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other covered employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a frontier developer from disciplining or terminating a covered employee for conduct that is unrelated to the claim of the retaliation.
(i)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(j)(1)This section does not impair or limit the applicability of Section 1102.5, including with respect to the rights of employees who are not covered employees to report violations of this chapter or Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
1107.2.The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
SEC. 5.(a)The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b)This act shall be liberally construed to effectuate its purposes.
(c)The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d)This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a frontier developer.
(e)This act shall not apply to the extent that it is preempted by federal law.
(f)This act preempts any rule, regulation, code, ordinance, or other law adopted by a city, county, city and county, municipality, or local agency on or after January 1, 2025, specifically related to the regulation of frontier developers with respect to their management of catastrophic risk.
SEC. 6.The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports, assessments of risks from internal use, and reports from covered employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research through companies large and small and through the states remarkable public and private universities.
(b)Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d)Targeted interventions to support effective artificial intelligence governance should balance the technologys benefits and the potential for material risks.
(e)Artificial intelligence developers have already voluntarily committed to creating safety and security protocols and releasing the results of risk assessments.(f)(e)In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(g)When(f)As industry actors conduct internal research on their technologies impacts, a significant information asymmetry can develop between those with privileged access to data and the broader public. public trust in these technologies would significantly benefit from access to information regarding, and increased awareness of, frontier AI capabilities.
(h)(g)Greater transparency, given current information deficits, transparency can also advance accountability, competition, and public trust.
(i)(h)Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(j)Adverse event(i)Incident reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(k)There is growing evidence that, unless(j)Unless they are developed with careful diligence and reasonable precaution, there is concern that advanced artificial intelligence systems could have capabilities that pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(l)(k)With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to the serious risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(l)While the major artificial intelligence developers have already voluntarily established the creation, use, and publication of frontier AI frameworks as an industry best practice, not all developers are providing reporting that is consistent and sufficient to ensure necessary transparency and protection of the public. Mandatory, standardized, and objective reporting by frontier developers is required to provide the government and the public with timely and accurate information.
(m)Timely reporting of critical safety incidents to the government is essential to ensure that public authorities are promptly informed of ongoing and emerging risks to public safety. This reporting enables the government to monitor, assess, and respond effectively in the event that advanced capabilities emerge in frontier artificial intelligence models that may pose a threat to the public.
(m)(n)In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(o)The recent release of the Governors California Report on Frontier AI Policy and testimony from legislative hearings on artificial intelligence before the Legislature reflect the advances in AI model capabilities that could pose potential catastrophic risk in frontier artificial intelligence, which this act aims to address.
(n)(p)It is the intent of the Legislature to create more transparency, but collective safety will depend in part on large frontier developers taking due care in their development and deployment of foundation frontier models proportional to the scale of the foreseeable risks.
SEC. 2.Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:25.1.Transparency in Frontier Artificial Intelligence Act22757.10.This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11.For purposes of this chapter:
(a)Affiliate means a person controlling, controlled by, or under common control with a specified person, directly or indirectly, through one or more intermediaries.
(a)(b)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(c)(1)Catastrophic risk means a foreseeable and material risk that a large frontier developers development, storage, use, or deployment of a foundation frontier model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident, scheme, or course of conduct incident involving a dangerous capability. frontier model doing any of the following:
(A)Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(B)Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if the conduct had been committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(C)Evading the control of its frontier developer or user.
(2)Catastrophic risk does not include a foreseeable and material risk from any of the following:
(A)Information that a frontier model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
(B)Lawful activity of the federal government.
(C)Harm caused by a frontier model in combination with other software if the frontier model did not materially contribute to the harm.
(c)(d)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model. frontier model that results in death or bodily injury.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation frontier model causing death, bodily injury, or damage to, or loss of, property. death or bodily injury.
(4)A foundation frontier model that uses deceptive techniques against the large frontier developer to subvert the controls or monitoring of its large frontier developer outside of the context of an evaluation designed to elicit this behavior. behavior and in a manner that demonstrates materially increased catastrophic risk.
(5)Attaining a dangerous capability or catastrophic risk threshold, as defined in the large developers safety and security protocol pursuant to paragraph (3) of subdivision (a) of Section 22757.12, for the first time.(d)Dangerous capability means the capacity of a foundation model to do any of the following:(1)Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.(2)Conduct or assist in a cyberattack.(3)Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.(4)Evade the control of its large developer or user.(e)(1)Deploy means to make a foundation frontier model available to a third party for use, modification, copying, or combination with other software.
(2)Deploy does not include making a foundation frontier model available to a third party for the primary purpose of developing or evaluating the foundation frontier model.
(f)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data set.
(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(g)Frontier AI framework means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
(h)Frontier developer means a person who has trained, or initiated the training of, a frontier model, with respect to which the person has used, or intends to use, at least as much computing power to train the frontier model as would meet the technical specifications found in subdivision (i).
(i)(1)Frontier model means a foundation model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
(2)The quantity of computing power described in paragraph (1) shall include computing for the original training run and for any subsequent fine-tuning, reinforcement learning, or other material modifications the developer applies to a preceding foundation model.
(g)(j)Large frontier developer means either of the following: a frontier developer that together with its affiliates collectively had annual gross revenues in excess of five hundred million dollars ($500,000,000) in the preceding calendar year.
(1)(A)Before January 1, 2027, large developer means a person who meets both of the following criteria:(i)(I)The person has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.(II)The quantity of computing power described in subclause (I) shall include computing for the original training run and any subsequent fine-tuning, reinforcement learning, or other material modifications to a preceding foundation model.(ii)The person had annual gross revenues in excess of five hundred million dollars ($500,000,000) in the preceding calendar year.(2)(A)Except as provided in subparagraph (B), on and after January 1, 2027, large developer has the meaning defined by a regulation adopted by the Attorney General pursuant to Section 22757.15.(B)If the Attorney General does not adopt a regulation described in subparagraph (A) by January 1, 2027, the definition in paragraph (1) shall be operative until the regulation is adopted.(h)(k)Model weight means a numerical parameter in a foundation frontier model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(i)(l)Property means tangible or intangible property.
(j)Safety and security protocol means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.22757.12.(a)A large frontier developer shall write, implement, comply with, and clearly and conspicuously publish on its internet website a safety and security protocol frontier AI framework that describes in specific detail applies to the large frontier developers frontier models and describes how the large frontier developer approaches all of the following:
(1)How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models are incapable of posing material catastrophic risks and which foundation models, if any, have been included.(2)The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.(3)(A)Thresholds used by the large developer to identify and assess whether a foundation model has attained a dangerous capability or poses a catastrophic risk.(B)How the large developer will assess whether a threshold described in subparagraph (A) has been attained, which may include multiple tiered thresholds for different dangerous capabilities or for catastrophic risk.(C)The actions the large developer will take if each threshold is attained.(4)The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.(5)The degree to which the large developers assessments of catastrophic risk and dangerous capabilities and the effectiveness of catastrophic risk mitigations are reproducible by external entities.(6)The extent to which, and how, a large developer will use third parties to assess catastrophic risks and dangerous capabilities and the effectiveness of mitigations of catastrophic risk.(7)The large developers cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.(8)To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, whether the large developer has the ability to promptly shut down copies of foundation models owned and controlled by the large developer, who the large developer will notify, and the timeline on which the large developer would take these steps.(9)The testing procedures that the large developer will use to assess and manage a catastrophic risk or dangerous capability resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to subdivision (d).(10)How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to subdivision (c).(1)Incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework.
(2)Defining and assessing thresholds used by the large frontier developer to identify and assess whether a frontier model has capabilities that could pose a catastrophic risk, which may include multiple-tiered thresholds.
(3)Applying mitigations to address the potential for catastrophic risks based on the results of assessments undertaken pursuant to paragraph (2).
(4)Reviewing assessments and adequacy of mitigations as part of the decision to deploy a frontier model or use it extensively internally.
(5)Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks.
(6)Revisiting and updating the frontier AI framework, including any criteria that trigger updates and how the large frontier developer determines when its frontier models are substantially modified enough to require disclosures pursuant to subdivision (c).
(7)Cybersecurity practices to secure unreleased model weights from unauthorized modification or transfer by internal or external parties.
(8)Identifying and responding to critical safety incidents.
(9)Instituting internal governance practices to ensure implementation of these processes.
(10)Assessing and managing catastrophic risk resulting from the internal use of its frontier models, including risks resulting from a frontier model circumventing oversight mechanisms.
(b)(1)A large frontier developer shall review and, as appropriate, update its frontier AI framework at least once per year.
(b)(2)If a large frontier developer makes a material modification to its safety and security protocol, frontier AI framework, the large frontier developer shall clearly and conspicuously publish the modified protocol frontier AI framework and a justification for that modification within 30 days.
(c)(1)Before, or concurrently with, deploying a new foundation frontier model or a substantially modified version of an existing foundation model that is covered by the large developers safety and security protocol, frontier model, a large frontier developer shall clearly and conspicuously publish on its internet website a transparency report containing both all of the following:
(1)(A)The results of any risk assessment or risk mitigation assessment conducted by the large developer or a third party that contracts with a large developer pursuant to its safety and security protocol, why the information gathered by the large developer or third party leads to the stated results, and the steps taken to address any identified risks.(B)A large developer shall disclose the time and extent of predeployment access provided to any third party described in subparagraph (A), whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third partys ability to disclose information about its assessment to the public or to government officials.(C)Whether a catastrophic risk threshold or dangerous capability threshold has been attained for the foundation model and any actions taken as a result.(2)(A)The reasoning behind the large developers decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.(B)A large developer may reuse an answer previously provided under subparagraph (A) if the rationale in question has not materially changed for the new deployment.(A)The internet website of the frontier developer.
(B)A mechanism that enables a natural person to communicate with the frontier developer.
(C)The release date of the frontier model.
(D)The languages supported by the frontier model.
(E)The modalities of output supported by the frontier model.
(F)The intended uses of the frontier model.
(G)Any generally applicable restrictions or conditions on uses of the frontier model.
(2)Before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, a large frontier developer shall include in the transparency report required by paragraph (1) summaries of all of the following:
(A)Assessments of catastrophic risks from the frontier model conducted pursuant to the large frontier developers frontier AI framework.
(B)The results of those assessments.
(C)The extent to which third-party evaluators were involved.
(D)Other steps taken to fulfill the requirements of the frontier AI framework with respect to the frontier model.
(3)A frontier developer that publishes the information described in paragraph (1) or (2) as part of a larger document, including a system card or model card, shall be deemed in compliance with the applicable paragraph.
(4)A frontier developer is encouraged, but not required, to make disclosures described in this subdivision that are consistent with, or superior to, industry best practices.
(d)A large frontier developer shall clearly and conspicuously publish on its internet website transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk or dangerous capabilities resulting from internal use of its foundation frontier models pursuant to the schedule the developer specifies in its safety and security protocol. every three months or pursuant to another reasonable schedule specified by the large frontier developer and communicated in writing to the Office of Emergency Services with written updates, as appropriate.
(e)(1)(A)A large frontier developer shall not make a materially false or misleading statement about catastrophic risk from its foundation models, frontier models or its management of catastrophic risk, or its implementation of or compliance with its safety and security protocol. risk.
(B)A large frontier developer shall not make a materially false or misleading statement about its implementation of, or compliance with, its frontier AI framework.
(2)This subdivision does not apply to a statement that was made in good faith and was reasonable under the circumstances.
(f)(1)When a large frontier developer publishes documents to comply with this section, the large frontier developer may make redactions to those documents that are necessary to protect the large frontier developers trade secrets, the large frontier developers cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2)If a large frontier developer redacts information in a document pursuant to this subdivision, the large frontier developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13.(a)The Attorney General Office of Emergency Services shall establish a mechanism to be used by a large frontier developer or a member of the public to report a critical safety incident that includes all of the following:
(1)The date of the critical safety incident.
(2)The reasons the incident qualifies as a critical safety incident.
(3)A short and plain statement describing the critical safety incident.
(4)Whether the incident was associated with internal use of a frontier model.
(b)(1)The Office of Emergency Services shall establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models.
(2)The Office of Emergency Services shall take all necessary precautions to limit access to any reports related to internal use of frontier models to only personnel with a specific need to know the information and to protect the reports from unauthorized access.
(b)(c)(1)Subject to paragraph (2), a large frontier developer shall report any critical safety incident pertaining to one or more of its foundation frontier models to the Attorney General Office of Emergency Services within 15 days of discovering the critical safety incident.
(2)If a large frontier developer discovers that a critical safety incident poses an imminent risk of death or serious physical injury, the large frontier developer shall disclose that incident within 24 hours to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and as required by law.
(3)A frontier developer that discovers information about a critical safety incident after filing the initial report required by this subdivision may file an amended report.
(4)A frontier developer is encouraged, but not required, to report critical safety incidents pertaining to foundation models that are not frontier models.
(c)(d)The Attorney General Office of Emergency Services shall review critical safety incident reports submitted by large frontier developers and may review reports submitted by members of the public.
(d)(e)(1)The Attorney General or the Office of Emergency Services may transmit reports of critical safety incidents and reports from covered employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code to the Legislature, the Governor, the federal government, or appropriate state agencies.
(2)The Attorney General or the Office of Emergency Services shall strongly consider any risks related to trade secrets, public safety, cybersecurity of a frontier developer, or national security when transmitting reports.
(e)(f)A report of a critical safety incident submitted to the Attorney General Office of Emergency Services pursuant to this section section, a report of assessments of catastrophic risk from internal use pursuant to Section 22757.12, and an a covered employee report made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(f)(g)(1)Beginning January 1, 2027, and annually thereafter, the Attorney General Office of Emergency Services shall produce a report with anonymized and aggregated information about critical safety incidents and reports from employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code that have been reviewed by the Attorney General Office of Emergency Services since the preceding report.
(2)The Attorney General Office of Emergency Services shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a large frontier developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Attorney General Office of Emergency Services shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795, and to the Governor.
(h)The Office of Emergency Services may adopt regulations designating one or more federal laws, regulations, or guidance documents that meet all of the following conditions for the purposes of subdivision (i):
(1)(A)The law, regulation, or guidance document imposes or states standards or requirements for critical safety incident reporting that are substantially equivalent to, or stricter than, those required by this section.
(B)The law, regulation, or guidance document described in subparagraph (A) does not need to require critical safety incident reporting to the State of California.
(2)The law, regulation, or guidance document is intended to assess, detect, or mitigate the catastrophic risk.
(i)(1)A frontier developer that intends to comply with this section by complying with the requirements of, or meeting the standards stated by, a federal law, regulation, or guidance document designated pursuant to subdivision (h) shall declare its intent to do so to the Office of Emergency Services.
(2)After a frontier developer has declared its intent pursuant to paragraph (1), both of the following apply:
(A)The frontier developer shall be deemed in compliance with this section to the extent that the frontier developer meets the standards of, or complies with the requirements imposed or stated by, the designated federal law, regulation, or guidance document until the frontier developer declares the revocation of that intent to the Office of Emergency Services or the Office of Emergency Services revokes a relevant regulation pursuant to subdivision (j).
(B)The failure by a frontier developer to meet the standards of, or comply with the requirements stated by, the federal law, regulation, or guidance document designated pursuant to subdivision (h) shall constitute a violation of this chapter.
(j)The Office of Emergency Services shall revoke a regulation adopted under subdivision (h) if the requirements of subdivision (h) are no longer met.
22757.14.(a)On or before January 1, 2027, and annually thereafter, the Attorney General may adopt regulations Department of Technology shall assess recent evidence and developments relevant to the purposes of this chapter and shall make recommendations about whether and how to update the definition of a large developer any of the following definitions for the purposes of this chapter to ensure that it they accurately reflects reflect technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development. standards:
(1)Frontier model so that it applies to foundation models at the frontier of artificial intelligence development.
(2)Frontier developer so that it applies to developers of frontier models who are themselves at the frontier of artificial intelligence development.
(3)Large frontier developer so that it applies to well-resourced frontier developers.
(b)In developing regulations making recommendations pursuant to this section, the Attorney General Department of Technology shall take into account all of the following:
(1)Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk. risk and shall align with a definition adopted in a federal law or regulation to the extent that it is consistent with the purposes of this chapter.
(2)Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3)The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations definition as a frontier developer or as a large frontier developer with an aim toward allowing earlier determinations if possible.
(4)The complexity of determining whether a person or foundation model is covered, with an aim toward allowing simpler determinations if possible.
(5)The external verifiability of determining whether a person or foundation model is covered, with an aim toward definitions that are verifiable by parties other than the large frontier developer.
(c)If the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General shall promptly submit a report to the Legislature, pursuant to Section 9795, with a proposal for managing this source of catastrophic risk but shall not include those developers within the definition of large developer without authorization in subsequently enacted legislation.(c)Upon developing recommendations pursuant to this section, the Department of Technology shall submit a report to the Legislature, pursuant to Section 9795 of the Government Code, with those recommendations.
(d)(1)Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about reports from covered employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code that have been reviewed by the Attorney General since the preceding report.
(2)The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a frontier developer, confidentiality of a covered employee, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Attorney General shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795 of the Government Code, and to the Governor.
22757.15.(a)A large frontier developer who violates this chapter that fails to publish or transmit a compliant document required to be published or transmitted under this chapter, makes a statement in violation of subdivision (e) of Section 22757.12, fails to report an incident as required by Section 22757.13, or fails to comply with its own frontier AI framework shall be subject to a civil penalty subject to all of the following: in an amount dependent upon the severity of the violation that does not exceed one million dollars ($1,000,000) per violation.
(1)(A)For an unknowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).(B)(i)If the violation is a first violation subject to a civil penalty pursuant to subparagraph (A), a large developer shall be provided with a 30-day period to cure the violation after notification by the Attorney General. If the violation is cured within that period, a civil penalty shall not be imposed for that violation.(ii)For the purposes of this subparagraph, a violation involving the missing or late publication or submission of a document is cured when the developer publishes or submits that document, even if the publication or submission occurs after the statutory deadline.(2)For a knowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk or an unknowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one hundred thousand dollars ($100,000).(3)For a knowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one million dollars ($1,000,000) for a violation that is the large developers first such violation and in an amount not exceeding ten million dollars ($10,000,000) for any subsequent violation.(b)A civil penalty described in this section shall be recovered in a civil action brought only by the Attorney General.
22757.16.The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
SEC. 3.Section 11546.8 is added to the Government Code, to read:11546.8.(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1)Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2)Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3)Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4)Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Catastrophic Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.(b)Catastrophic risk has the meaning defined in Section 22757.11 of the Business and Professions Code.(a)(1)Catastrophic risk means a foreseeable and material risk that a frontier developers development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident involving a foundation model doing any of the following:
(A)Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(B)Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(C)Evading the control of its frontier developer or user.
(2)Catastrophic risk does not include a foreseeable and material risk from any of the following:
(A)Information that a foundation model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
(B)Lawful activity of the federal government.
(C)Harm caused by a foundation model in combination with other software where the foundation model did not materially contribute to the harm.
(b)Covered employee means an employee responsible for assessing, managing, or addressing risk of critical safety incidents.
(c)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of the model weights of a foundation model that results in death, bodily injury, or damage to, or loss of, property.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation model causing death or bodily injury.
(4)A foundation model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.
(d)Foundation model has the meaning defined in Section 22757.11 of the Business and Professions Code.
(e)Frontier developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(c)(f)Large frontier developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(d)Employee means a person who performs services for an employer, including both of the following:(1)A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing catastrophic risk, including all of the following:(A)An independent contractor.(B)A freelance worker.(C)A person employed by a labor contractor.(D)A board member.(2)Corporate officers.(e)Foundation model has the same meaning as defined in Section 22757.11 of the Business and Professions Code.(f)Labor contractor means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employers usual course of business.1107.1.(a)A large frontier developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents an a covered employee from disclosing, or retaliates against an a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses either of the following:
(1)The large frontier developers activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk.
(2)The large frontier developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b)A large frontier developer shall not enter into a contract that prevents an a covered employee from making a disclosure protected under Section 1102.5.
(c)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:(1)The large developers activities pose a catastrophic risk.(2)The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.(d)An(c)A covered employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(e)(d)A large frontier developer shall provide a clear notice to all covered employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the large frontier developer a notice to all covered employees of their rights under this section, ensuring that any new covered employee receives equivalent notice, and ensuring that any covered employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each covered employee of the covered employees rights under this section and ensuring that the notice is received and acknowledged by all of those covered employees.
(f)(e)(1)A large frontier developer shall provide a reasonable internal process through which an a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developers activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large frontier developers investigation of the disclosure and the actions taken by the large frontier developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large frontier developer at least once each quarter.
(B)If an a covered employee has alleged wrongdoing by an officer or director of the large frontier developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(g)(f)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(h)(g)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the covered employee, the large frontier developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the covered employee had not engaged in activities protected by this section.
(i)(h)(1)In a civil action or administrative proceeding brought pursuant to this section, an a covered employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other covered employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a large frontier developer from disciplining or terminating an a covered employee for conduct that is unrelated to the claim of the retaliation.
(j)(i)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k)(j)(1)This section does not impair or limit the applicability of Section 1102.5. 1102.5, including with respect to the rights of employees who are not covered employees to report violations of this chapter or Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
1107.2.The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
SEC. 5.(a)The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b)This act shall be liberally construed to effectuate its purposes.
(c)The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d)This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a large frontier developer.
(e)This act shall not apply to the extent that it is preempted by federal law.
(f)This act preempts any rule, regulation, code, ordinance, or other law adopted by a city, county, city and county, municipality, or local agency on or after January 1, 2025, specifically related to the regulation of frontier developers with respect to their management of catastrophic risk.
SEC. 6.The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports reports, assessments of risks from internal use, and reports from covered employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research through companies large and small and through the states remarkable public and private universities.
(b)Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d)Targeted interventions to support effective artificial intelligence governance should balance the technologys benefits and material risks.
(e)Artificial intelligence developers have already voluntarily committed to creating safety and security protocols and releasing the results of risk assessments.
(f)In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(g)When industry actors conduct internal research on their technologies impacts, a significant information asymmetry can develop between those with privileged access to data and the broader public.
(h)Greater transparency, given current information deficits, can advance accountability, competition, and public trust.
(i)Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(j)Adverse event reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(k)There is growing evidence that, unless they are developed with careful diligence and reasonable precaution, advanced artificial intelligence systems could pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(l)With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to the risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(m)In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(n)It is the intent of the Legislature to create more transparency, but collective safety will depend in part on large developers taking due care in their development and deployment of foundation models proportional to the scale of the foreseeable risks.
SEC. 2.Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:25.1.Transparency in Frontier Artificial Intelligence Act22757.10.This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Catastrophic risk means a foreseeable and material risk that a large developers development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 50 people or more than one billion dollars ($1,000,000,000) in damage to, or loss of, property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
(c)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation model causing death, bodily injury, or damage to, or loss of, property.
(4)A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
(5)Attaining a dangerous capability or catastrophic risk threshold, as defined in the large developers safety and security protocol pursuant to paragraph (3) of subdivision (a) of Section 22757.12, for the first time.
(d)Dangerous capability means the capacity of a foundation model to do any of the following:
(1)Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(2)Conduct or assist in a cyberattack.
(3)Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
(4)Evade the control of its large developer or user.
(e)(1)Deploy means to make a foundation model available to a third party for use, modification, copying, or combination with other software.
(2)Deploy does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
(f)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data set.
(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(g)Large developer means either of the following:
(1)(A)Before January 1, 2027, large developer means a person who meets both of the following criteria:
(i)(I)The person has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
(II)The quantity of computing power described in subclause (I) shall include computing for the original training run and any subsequent fine-tuning, reinforcement learning, or other material modifications to a preceding foundation model.
(ii)The person had annual gross revenues in excess of one five hundred million dollars ($100,000,000) ($500,000,000) in the preceding calendar year.
(2)(A)Except as provided in subparagraph (B), on and after January 1, 2027, large developer has the meaning defined by a regulation adopted by the Attorney General pursuant to Section 22757.15.
(B)If the Attorney General does not adopt a regulation described in subparagraph (A) by January 1, 2027, the definition in paragraph (1) shall be operative until the regulation is adopted.
(h)Model weight means a numerical parameter in a foundation model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(i)Property means tangible or intangible property.
(j)Safety and security protocol means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
22757.12.(a)A large developer shall write, implement, comply with, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
(1)How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models are incapable of posing material catastrophic risks and which foundation models, if any, have been included.
(2)The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
(3)(A)Thresholds used by the large developer to identify and assess whether a foundation model has attained a dangerous capability or poses a catastrophic risk.
(B)How the large developer will assess whether a threshold described in subparagraph (A) has been attained, which may include multiple tiered thresholds for different dangerous capabilities or for catastrophic risk.
(C)The actions the large developer will take if each threshold is attained.
(4)The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
(5)The degree to which the large developers assessments of catastrophic risk and dangerous capabilities and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
(6)The extent to which, and how, a large developer will use third parties to assess catastrophic risks and dangerous capabilities and the effectiveness of mitigations of catastrophic risk.
(7)The large developers cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
(8)To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, whether the large developer has the ability to promptly shut down copies of foundation models owned and controlled by the large developer, who the large developer will notify, and the timeline on which the large developer would take these steps.
(9)The testing procedures that the large developer will use to assess and manage a catastrophic risk or dangerous capability resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to subdivision (d).
(10)How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to subdivision (c).
(b)If a large developer makes a material modification to its safety and security protocol, the large developer shall clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
(c)Before, or concurrently with, deploying a new foundation model or a substantially modified version of an existing foundation model that is covered by the large developers safety and security protocol, a large developer shall clearly and conspicuously publish on its internet website a transparency report containing both of the following:
(1)(A)The results of any risk assessment or risk mitigation assessment conducted by the large developer or a third party that contracts with a large developer pursuant to its safety and security protocol, why the information gathered by the large developer or third party leads to the stated results, and the steps taken to address any identified risks.
(B)A large developer shall disclose the time and extent of predeployment access provided to any third party described in subparagraph (A), whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third partys ability to disclose information about its assessment to the public or to government officials.
(C)Whether a catastrophic risk threshold or dangerous capability threshold has been attained for the foundation model and any actions taken as a result.
(2)(A)The reasoning behind the large developers decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
(B)A large developer may reuse an answer previously provided under subparagraph (A) if the rationale in question has not materially changed for the new deployment.
(d)A large developer shall clearly and conspicuously publish on its internet website any assessment of catastrophic risk or dangerous capabilities resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
(e)A large developer shall not make a materially false or misleading statement about catastrophic risk from its foundation models, its management of catastrophic risk, or its implementation of or compliance with its safety and security protocol.
(f)(1)When a large developer publishes documents to comply with this section, the large developer may make redactions to those documents that are necessary to protect the large developers trade secrets, the large developers cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2)If a large developer redacts information in a document pursuant to this subdivision, the large developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13.(a)The Attorney General shall establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
(1)The date of the critical safety incident.
(2)The reasons the incident qualifies as a critical safety incident.
(3)A short and plain statement describing the critical safety incident.
(b)(1)Subject to paragraph (2), a large developer shall report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
(2)If a large developer discovers that a critical safety incident poses an imminent risk of death or serious physical injury, the large developer shall disclose that incident within 24 hours to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and as required by law.
(c)The Attorney General shall review critical safety incident reports submitted by large developers and may review reports submitted by members of the public.
(d)The Attorney General may transmit reports of critical safety incidents, incidents and reports from employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor reports required by Section 22757.14 Code to the Legislature, the Governor, the federal government, or appropriate state agencies.
(e)A report of a critical safety incident submitted to the Attorney General pursuant to this section, a section and an employee report made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and a summary of an auditors report required by Section 22757.14 Code are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(f)(1)Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about critical safety incidents, incidents and reports from employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditors reports required by Section 22757.14 Code that have been reviewed by the Attorney General since the preceding report.
(2)The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a large developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Attorney General shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795, and to the Governor.
22757.14.(a)Beginning January 1, 2030, and at least annually thereafter, a large developer shall retain an independent third-party auditor to produce a report assessing both of the following:(1)Whether the large developer has substantially complied with its safety and security protocol and any instances of substantial noncompliance during the prior year.(2)Any instances in which the large developers safety and security protocol has not been stated clearly enough to determine whether the large developer has complied.(b)A large developer shall allow the third-party auditor access to all materials produced to comply with this chapter and any other materials reasonably necessary to perform the assessment required by this section.(c)A large developer shall retain a report required by this section for five years.(d)In conducting an audit for a large developer pursuant to this section, an auditor shall employ or contract one or more individuals with expertise in corporate compliance and one or more individuals with technical expertise in the safety of foundation models.(e)Within 30 days after completing an audit, the auditor shall transmit to the Attorney General a high-level summary of the report required by this section that fairly presents, in all material respects, the outcome of the audit.(f)An auditor shall not knowingly include a material misrepresentation or omit a material fact in a summary or report produced pursuant to this section.22757.15.22757.14.(a)On or before January 1, 2027, and annually thereafter, the Attorney General may adopt regulations to update the definition of a large developer for the purposes of this chapter to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development.
(b)In developing regulations pursuant to this section, the Attorney General shall take into account all of the following:
(1)Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
(2)Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3)The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
(4)The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
(5)The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
(c)If the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General shall promptly submit a report to the Legislature, pursuant to Section 9795, with a proposal for managing this source of catastrophic risk but shall not include those developers within the definition of large developer without authorization in subsequently enacted legislation.
22757.16.22757.15.(a)A large developer who violates this chapter shall be subject to a civil penalty subject to all of the following:
(1)(A)For an unknowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(B)(i)If the violation is a first violation subject to a civil penalty pursuant to subparagraph (A), a large developer shall be provided with a 30-day period to cure the violation after notification by the Attorney General. If the violation is cured within that period, a civil penalty shall not be imposed for that violation.
(ii)For the purposes of this subparagraph, a violation involving the missing or late publication or submission of a document is cured when the developer publishes or submits that document, even if the publication or submission occurs after the statutory deadline.
(2)For a knowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk or an unknowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one hundred thousand dollars ($100,000).
(3)For a knowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one million dollars ($1,000,000) for a violation that is the large developers first such violation and in an amount not exceeding ten million dollars ($10,000,000) for any subsequent violation.
(b)A violation of this chapter by an auditor shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).(c)(b)A civil penalty described in this section shall be recovered in a civil action brought only by the Attorney General.
SEC. 3.Section 11546.8 is added to the Government Code, to read:11546.8.(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1)Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2)Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3)Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4)Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Catastrophic Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Catastrophic risk has the meaning defined in Section 22757.11 of the Business and Professions Code.
(c)Large developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(d)Employee means a person who performs services for an employer, including both of the following:
(1)A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing catastrophic risk, including all of the following:
(A)An independent contractor.
(B)A freelance worker.
(C)A person employed by a labor contractor.
(D)A board member.
(2)Corporate officers.
(e)Foundation model has the same meaning as defined in Section 22757.11 of the Business and Professions Code.
(f)Labor contractor means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employers usual course of business.
1107.1.(a)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1)The large developers activities pose a catastrophic risk.
(2)The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b)A large developer shall not enter into a contract that prevents an employee from making a disclosure protected under Section 1102.5.
(c)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
(1)The large developers activities pose a catastrophic risk.
(2)The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(d)An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(e)A large developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each employee of the employees rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(f)(1)A large developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developers activities present a catastrophic risk or that the large developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large developers investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large developer at least once each quarter.
(B)If an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(g)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(h)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the large developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(i)(1)In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(j)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k)(1)This section does not impair or limit the applicability of Section 1102.5.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
SEC. 5.(a)The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b)This act shall be liberally construed to effectuate its purposes.
(c)The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d)This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a large developer.
(e)This act shall not apply to the extent that it is preempted by federal law.
SEC. 6.The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports, summaries of auditors reports, reports and reports from employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research through companies large and small and through the states remarkable public and private universities.
(b)Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d)Targeted interventions to support effective artificial intelligence governance should balance the technologys benefits and material risks.
(e)Artificial intelligence developers have already voluntarily committed to creating safety and security protocols and releasing the results of risk assessments.
(f)In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(g)When industry actors conduct internal research on their technologies impacts, a significant information asymmetry can develop between those with privileged access to data and the broader public.
(h)Greater transparency, given current information deficits, can advance accountability, competition, and public trust.
(i)Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(j)Adverse event reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(k)There is growing evidence that, unless they are developed with careful diligence and reasonable precaution, advanced artificial intelligence systems could pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(l)With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to the risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(m)A computational threshold of 10^26 floating point operations captures the current frontier of foundation model development and captures only highly resourced developers spending hundreds of millions of dollars to develop foundation models.(n)(m)In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(o)(n)It is the intent of the Legislature to create more transparency, but collective safety will depend in part on large developers taking due care in their development and deployment of foundation models proportional to the scale of the foreseeable risks.
SEC. 2.Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:25.1.Transparency in Frontier Artificial Intelligence Act22757.10.This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Catastrophic risk means a foreseeable and material risk that a large developers development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 100 50 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or to, or loss of, property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
(c)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation model causing death, bodily injury, or damage to rights in money or to, or loss of, property.
(4)A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
(5)Attaining a dangerous capability or catastrophic risk threshold, as defined in the large developers safety and security protocol pursuant to paragraph (3) of subdivision (a) of Section 22757.12, for the first time.
(d)Dangerous capability means the capacity of a foundation model to do any of the following:
(1)Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(2)Conduct or assist in a cyberattack.
(3)Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of extortion, theft, including theft by deception, or a serious or violent felony. murder, assault, extortion, or theft, including theft by false pretense.
(4)Evade the control of its large developer or user.
(e)(1)Deploy means to make a foundation model available to a third party for use, modification, copying, or combination with other software.
(2)Deploy does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
(f)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data set.
(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(f)(g)Large developer means either of the following:
(1)(A)Before January 1, 2027, large developer means a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations. meets both of the following criteria:
(i)(I)The person has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
(B)(II)The quantity of computing power described in subparagraph (A) subclause (I) shall include computing for the original training run and any subsequent fine-tuning, reinforcement learning, or other material modifications to a preceding foundation model.
(ii)The person had annual gross revenues in excess of one hundred million dollars ($100,000,000) in the preceding calendar year.
(2)(A)Except as provided in subparagraph (B), on and after January 1, 2027, large developer has the meaning defined by a regulation adopted by the Attorney General pursuant to Section 22757.15.
(B)If the Attorney General does not adopt a regulation described in subparagraph (A) by January 1, 2027, the definition in paragraph (1) shall be operative until the regulation is adopted.
(g)Foundation model means an artificial intelligence model that is all of the following:(1)Trained on a broad data set.(2)Designed for generality of output.(3)Adaptable to a wide range of distinctive tasks.(h)Model weight means a numerical parameter in a foundation model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(i)Property means tangible or intangible property.
(i)(j)Safety and security protocol means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
22757.12.(a)A large developer shall write, implement, comply with, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
(1)How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models do not pose are incapable of posing material catastrophic risks. risks and which foundation models, if any, have been included.
(2)The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
(3)(A)Thresholds used by the large developer to identify and assess whether a foundation model has attained a dangerous capability or poses a catastrophic risk.
(B)How the large developer will assess whether a threshold described in subparagraph (A) has been attained, which may include multiple tiered thresholds for different dangerous capabilities or for catastrophic risk.
(C)The actions the large developer will take if each threshold is attained.
(3)(4)The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
(4)(5)The degree to which the large developers assessments of catastrophic risk and dangerous capabilities and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
(5)(6)The extent to which, and how, a large developer will use third parties to assess catastrophic risks and dangerous capabilities and the effectiveness of mitigations of catastrophic risk.
(6)(7)The large developers cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
(7)(8)To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, whether the large developer has the ability to promptly shut down copies of foundation models owned and controlled by the large developer, who the large developer will notify notify, and the timeline on which the large developer would take these steps.
(8)(9)The testing procedures that the large developer will use to assess and manage a catastrophic risk or dangerous capability resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to subdivision (d).
(9)(10)How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to subdivision (c).
(b)If a large developer makes a material modification to its safety and security protocol, the large developer shall clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
(c)Before Before, or concurrently with with, deploying a new foundation model or a substantially modified version of an existing foundation model, model that is covered by the large developers safety and security protocol, a large developer shall clearly and conspicuously publish on its internet website a transparency report containing all both of the following:
(1)The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted by the large developer pursuant to its safety and security protocol.(2)(A)The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted pursuant to a large developers safety and security protocol that is conducted by a third party that contracts with a large developer.(1)(A)The results of any risk assessment or risk mitigation assessment conducted by the large developer or a third party that contracts with a large developer pursuant to its safety and security protocol, why the information gathered by the large developer or third party leads to the stated results, and the steps taken to address any identified risks.
(B)A large developer shall disclose the time and extent of predeployment access provided to any third party described in subparagraph (A), whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third partys ability to disclose information about its assessment to the public or to government officials.
(C)Whether a catastrophic risk threshold or dangerous capability threshold has been attained for the foundation model and any actions taken as a result.
(3)(2)(A)If the deployment would pose a catastrophic risk, the The reasoning behind the large developers decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
(B)A large developer may reuse an answer previously provided under subparagraph (A) if the rationale in question has not materially changed for the new deployment.
(d)A large developer shall clearly and conspicuously publish on its internet website any assessment of catastrophic risk or dangerous capabilities resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
(e)A large developer shall not make a materially false or misleading statement about catastrophic risk from its foundation models or models, its management of catastrophic risk. risk, or its implementation of or compliance with its safety and security protocol.
(f)(1)When a large developer publishes documents to comply with this section, the large developer may make redactions to those documents that are necessary to protect the large developers trade secrets, the large developers cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2)If a large developer redacts information in a document pursuant to this subdivision, the large developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13.(a)The Attorney General shall establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
(1)The date of the critical safety incident.
(2)The reasons the incident qualifies as a critical safety incident.
(3)A short and plain statement describing the critical safety incident.
(b)A (1)Subject to paragraph (2), a large developer shall report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
(2)If a large developer discovers that a critical safety incident poses an imminent risk of death or serious physical injury, the large developer shall disclose that incident within 24 hours to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and as required by law.
(c)The Attorney General shall review critical safety incident reports submitted by large developers and may review reports submitted by members of the public.
(d)The Attorney General may transmit reports of critical safety incidents, reports from employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor reports required by Section 22757.14 to the Legislature, the Governor, the federal government, or appropriate state agencies.
(e)A report of a critical safety incident submitted to the Attorney General pursuant to this section, a employee report made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and a summary of an auditors report required by Section 22757.14 are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(f)(1)Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about critical safety incidents, reports from employees made pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditors reports required by Section 22757.14 that have been reviewed by the Attorney General since the preceding report.
(2)The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a large developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3)The Attorney General shall transmit a report pursuant to this subdivision to the Legislature, pursuant to Section 9795, and to the Governor.
22757.14.(a)A violation of this chapter shall be subject to a civil penalty in an amount not to exceed ____ dollars ($____) per violation, unless the violation is willful or reckless, in which case the civil penalty shall be in an amount not to exceed ____ dollars ($____).(b)The civil penalty shall be assessed in a civil action brought only by the Attorney General.22757.14.(a)Beginning January 1, 2030, and at least annually thereafter, a large developer shall retain an independent third-party auditor to produce a report assessing both of the following:
(1)Whether the large developer has substantially complied with its safety and security protocol and any instances of substantial noncompliance during the prior year.
(2)Any instances in which the large developers safety and security protocol has not been stated clearly enough to determine whether the large developer has complied.
(b)A large developer shall allow the third-party auditor access to all materials produced to comply with this chapter and any other materials reasonably necessary to perform the assessment required by this section.
(c)A large developer shall retain a report required by this section for five years.
(d)In conducting an audit for a large developer pursuant to this section, an auditor shall employ or contract one or more individuals with expertise in corporate compliance and one or more individuals with technical expertise in the safety of foundation models.
(e)Within 30 days after completing an audit, the auditor shall transmit to the Attorney General a high-level summary of the report required by this section that fairly presents, in all material respects, the outcome of the audit.
(f)An auditor shall not knowingly include a material misrepresentation or omit a material fact in a summary or report produced pursuant to this section.
22757.15.(a)On or before January 1, 2027, and annually thereafter, the Attorney General may adopt regulations to update the definition of a large developer for the purposes of this chapter to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development.
(b)In developing regulations pursuant to this section, the Attorney General shall take into account all of the following:
(1)Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
(2)Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3)The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
(4)The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
(5)The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
(c)If the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General shall promptly submit a report to the Legislature, pursuant to Section 9795, with a proposal for managing this source of catastrophic risk but shall not include those developers within the definition of large developer without authorization in subsequently enacted legislation.
22757.16.(a)A large developer who violates this chapter shall be subject to a civil penalty subject to all of the following:
(1)(A)For an unknowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(B)(i)If the violation is a first violation subject to a civil penalty pursuant to subparagraph (A), a large developer shall be provided with a 30-day period to cure the violation after notification by the Attorney General. If the violation is cured within that period, a civil penalty shall not be imposed for that violation.
(ii)For the purposes of this subparagraph, a violation involving the missing or late publication or submission of a document is cured when the developer publishes or submits that document, even if the publication or submission occurs after the statutory deadline.
(2)For a knowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk or an unknowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one hundred thousand dollars ($100,000).
(3)For a knowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one million dollars ($1,000,000) for a violation that is the large developers first such violation and in an amount not exceeding ten million dollars ($10,000,000) for any subsequent violation.
(b)A violation of this chapter by an auditor shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(c)A civil penalty described in this section shall be recovered in a civil action brought only by the Attorney General.
SEC. 3.Section 11546.8 is added to the Government Code, to read:11546.8.(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1)Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2)Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3)Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4)Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Catastrophic Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Catastrophic risk has the meaning defined in Section 22757.11 of the Business and Professions Code.
(c)Large developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(d)Employee means a person who performs services for an employer, including both of the following:
(1)A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing catastrophic risk, including all of the following:
(A)An independent contractor.
(B)A freelance worker.
(C)A person employed by a labor contractor.
(D)A board member.
(2)Corporate officers.
(e)Foundation model has the same meaning as defined in Section 22757.11 of the Business and Professions Code.
(f)Labor contractor means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employers usual course of business.
1107.1.(a)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1)The large developers activities pose a catastrophic risk.
(2)The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b)A large developer shall not enter into a contract that prevents an employee from making a disclosure protected under Section 1102.5.
(c)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
(1)The large developers activities pose a catastrophic risk.
(2)The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(d)An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(e)A large developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each employee of the employees rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(f)(1)A large developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developers activities present a catastrophic risk or that the large developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large developers investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large developer at least once each quarter.
(B)If an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(g)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(h)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the large developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(i)(1)In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(j)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k)(1)This section does not impair or limit the applicability of Section 1102.5.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
SEC. 5.(a)The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b)This act shall be liberally construed to effectuate its purposes.
(c)The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d)This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a large developer.
(e)This act shall not apply to the extent that it is preempted by federal law.
SEC. 6.The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports, summaries of auditors reports, and reports from employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research through companies large and small and through the states remarkable public and private universities.
(b)Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d)Targeted interventions to support effective artificial intelligence governance should balance the technologys benefits and material risks.
(e)Artificial intelligence developers have already voluntarily committed to creating safety and security protocols and releasing the results of risk assessments.
(f)In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(g)When industry actors conduct internal research on their technologies impacts, a significant information asymmetry can develop between those with privileged access to data and the broader public.
(h)Greater transparency, given current information deficits, can advance accountability, competition, and public trust.
(i)Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(j)Adverse event reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(k)There is growing evidence that, unless they are developed with careful diligence and reasonable precaution, advanced artificial intelligence systems could pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(l)With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to the risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(m)A computational threshold of 10^26 floating point operations captures the current frontier of foundation model development and captures only highly resourced developers spending hundreds of millions of dollars to develop foundation models.
(n)In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(o)It is the intent of the Legislature to create more transparency, but collective safety will depend in part on large developers taking due care in their development and deployment of foundation models proportional to the scale of the foreseeable risks.
SEC. 2.Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:25.1.Transparency in Frontier Artificial Intelligence Act22757.10.This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Catastrophic risk means a foreseeable and material risk that a large developers development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
(c)Critical safety incident means any of the following:
(1)Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
(2)Harm resulting from the materialization of a catastrophic risk.
(3)Loss of control of a foundation model causing death, bodily injury, or damage to rights in money or property.
(4)A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
(d)Dangerous capability means the capacity of a foundation model to do any of the following:
(1)Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(2)Conduct or assist in a cyberattack.
(3)Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of extortion, theft, including theft by deception, or a serious or violent felony.
(4)Evade the control of its large developer or user.
(e)(1)Deploy means to make a foundation model available to a third party for use, modification, copying, or combination with other software.
(2)Deploy does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
(f)Large developer means either of the following:
(1)(A)Before January 1, 2027, large developer means a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
(B)The quantity of computing power described in subparagraph (A) shall include computing for the original training run and any subsequent fine-tuning, reinforcement learning, or other material modifications to a preceding foundation model.
(2)(A)Except as provided in subparagraph (B), on and after January 1, 2027, large developer has the meaning defined by a regulation adopted by the Attorney General pursuant to Section 22757.15.
(B)If the Attorney General does not adopt a regulation described in subparagraph (A) by January 1, 2027, the definition in paragraph (1) shall be operative until the regulation is adopted.
(g)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data set.
(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(h)Model weight means a numerical parameter in a foundation model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(i)Safety and security protocol means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
22757.12.(a)A large developer shall write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
(1)How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models do not pose material catastrophic risks.
(2)The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
(3)The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
(4)The degree to which the large developers assessments of catastrophic risk and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
(5)The extent to which, and how, a large developer will use third parties to assess catastrophic risks and the effectiveness of mitigations of catastrophic risk.
(6)The large developers cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
(7)To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, who the large developer will notify and the timeline on which the large developer would take these steps.
(8)The testing procedures that the large developer will use to assess and manage a catastrophic risk resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to subdivision (d).
(9)How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to subdivision (c).
(b)If a large developer makes a material modification to its safety and security protocol, the large developer shall clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
(c)Before or concurrently with deploying a new foundation model or a substantially modified version of an existing foundation model, a large developer shall clearly and conspicuously publish on its internet website a transparency report containing all of the following:
(1)The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted by the large developer pursuant to its safety and security protocol.
(2)(A)The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted pursuant to a large developers safety and security protocol that is conducted by a third party that contracts with a large developer.
(B)A large developer shall disclose the time and extent of predeployment access provided to any third party described in subparagraph (A), whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third partys ability to disclose information about its assessment to the public or to government officials.
(3)(A)If the deployment would pose a catastrophic risk, the reasoning behind the large developers decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
(B)A large developer may reuse an answer previously provided under subparagraph (A) if the rationale in question has not materially changed for the new deployment.
(d)A large developer shall clearly and conspicuously publish on its internet website any assessment of catastrophic risk resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
(e)A large developer shall not make a materially false or misleading statement about catastrophic risk from its foundation models or its management of catastrophic risk.
(f)(1)When a large developer publishes documents to comply with this section, the large developer may make redactions to those documents that are necessary to protect the large developers trade secrets, the large developers cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2)If a large developer redacts information in a document pursuant to this subdivision, the large developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13.(a)The Attorney General shall establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
(1)The date of the critical safety incident.
(2)The reasons the incident qualifies as a critical safety incident.
(3)A short and plain statement describing the critical safety incident.
(b)A large developer shall report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
(c)The Attorney General shall review critical safety incident reports submitted by large developers and may review reports submitted by members of the public.
22757.14.(a)A violation of this chapter shall be subject to a civil penalty in an amount not to exceed ____ dollars ($____) per violation, unless the violation is willful or reckless, in which case the civil penalty shall be in an amount not to exceed ____ dollars ($____).
(b)The civil penalty shall be assessed in a civil action brought only by the Attorney General.
22757.15.(a)On or before January 1, 2027, and annually thereafter, the Attorney General may adopt regulations to update the definition of a large developer for the purposes of this chapter to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development.
(b)In developing regulations pursuant to this section, the Attorney General shall take into account all of the following:
(1)Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
(2)Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3)The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
(4)The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
(5)The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
(c)If the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General shall promptly submit a report to the Legislature, pursuant to Section 9795, with a proposal for managing this source of catastrophic risk but shall not include those developers within the definition of large developer without authorization in subsequently enacted legislation.
SECTION 1.Section 11547.6.1 is added to the Government Code, to read:11547.6.1.SEC. 3.Section 11546.8 is added to the Government Code, to read:11546.8.(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following: as follows:
(A)Representatives(1)Four representatives of the University of California and other public and private academic research institutions and national laboratories. laboratories appointed by the Secretary of Government Operations.
(B)Representatives(2)Three representatives of impacted workforce labor organizations. organizations appointed by the Speaker of the Assembly.
(C)Representatives(3)Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates. advocates appointed by the Senate Rules Committee.
(D)Experts(4)Four experts in technology and artificial intelligence to provide technical assistance. assistance appointed by the Secretary of Government Operations.
(E)Personnel from other relevant departments and agencies, as necessary.(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 2.SEC. 4.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Critical Catastrophic Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Critical Catastrophic risk means a foreseeable and material risk that a developers development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property, through any of the following: has the meaning defined in Section 22757.11 of the Business and Professions Code.
(1)The creation and release of a chemical, biological, radiological, or nuclear weapon.(2)A cyberattack.(3)A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.(4)A foundation model evading the control of its developer or user.(c)Developer means a person that has trained at least one foundation model with a quantity of computational power that costs at least one hundred million dollars ($100,000,000) when measured using prevailing market prices of cloud compute.(c)Large developer has the meaning defined in Section 22757.11 of the Business and Professions Code.
(d)Employee means a person who performs services for an employer, including both of the following:
(1)A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing critical catastrophic risk, including all of the following:
(A)An independent contractor.
(B)A freelance worker.
(C)A person employed by a labor contractor.
(D)A board member.
(2)Corporate officers.
(e)Foundation model means an artificial intelligence model that is all of the following: has the same meaning as defined in Section 22757.11 of the Business and Professions Code.
(1)Trained on a broad data set.(2)Designed for generality of output.(3)Adaptable to a wide range of distinctive tasks.(f)Labor contractor means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employers usual course of business.
1107.1.(a)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1)The large developers activities pose a critical catastrophic risk.
(2)The large developer has made false or misleading statements about its management of critical risk. violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b)A large developer shall not enter into a contract that prevents an employee from making a disclosure protected under Section 1102.5.
(c)A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of critical catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
(1)The large developers activities pose a critical catastrophic risk.
(2)The large developer has made a false or misleading statement about its management of critical risk. violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(d)An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(e)A large developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each employee of the employees rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(f)(1)A large developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developers activities present a critical risk, catastrophic risk or that the large developer has made a false or misleading statement about its management of critical risk, violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large developers investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large developer at least once each quarter.
(B)If an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(g)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(h)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the large developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(i)(1)In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(j)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k)(1)This section does not impair or limit the applicability of Section 1102.5.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
SEC. 5.(a)The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b)This act shall be liberally construed to effectuate its purposes.
(c)The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d)This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a large developer.
(e)This act shall not apply to the extent that it is preempted by federal law.
(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following:
(A)Representatives of the University of California and other public and private academic research institutions and national laboratories.
(B)Representatives of impacted workforce labor organizations.
(C)Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
(D)Experts in technology and artificial intelligence to provide technical assistance.
(E)Personnel from other relevant departments and agencies, as necessary.
(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 2.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Critical Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Critical risk means a foreseeable and material risk that a developers development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property, through any of the following:
(1)The creation and release of a chemical, biological, radiological, or nuclear weapon.
(2)A cyberattack.
(3)A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
(4)A foundation model evading the control of its developer or user.
(c)Developer means a person that has trained at least one foundation model with a quantity of computational power that costs at least one hundred million dollars ($100,000,000) when measured using prevailing market prices of cloud compute.
(d)Employee means a person who performs services for wages or a salary under a contract of employment, express or implied, for an employer, including both of the following:
(1)(A)Contractors or subcontractors and unpaid advisors A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing the risk of critical harm from covered models and covered model derivatives. critical risk, including all of the following:
(B)As used in this paragraph, contractor or subcontractor means a firm, corporation, partnership, or association and its responsible managing officer, as well as any supervisors, managers, and officers found by the Labor Commissioner to be personally and substantially responsible for the willful violation of this chapter.(A)An independent contractor.
(B)A freelance worker.
(C)A person employed by a labor contractor.
(2)Corporate officers.
(e)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on a broad data. data set.
(2)Uses self-supervision in the training process.(3)Applicable across a wide range of contexts.(2)Designed for generality of output.
(3)Adaptable to a wide range of distinctive tasks.
(f)Labor contractor means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employers usual course of business.
1107.1.(a)A developer shall not make, adopt, or enforce enforce, or enter into a rule, regulation, or policy policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1)The developers activities pose a critical risk.
(2)The developer has made false or misleading statements about its management of critical risk.
(b)A developer shall not enter into a contract that prevents an employee from making a disclosure protected under Section 1102.5.
(c)A developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the developer related to the assessment, management, or addressing of critical risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
(1)The developers activities pose a critical risk.
(2)The developer has made a false or misleading statement about its management of critical risk.
(b)
(d)An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(c)
(e)A developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each employee of the employees rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(d)
(f)(1)A developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developers activities present a critical risk, or that the developer has made a false or misleading statement about its management of critical risk, including a monthly update to the person who made the disclosure regarding the status of the developers investigation of the disclosure and the actions taken by the developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the developer at least once each quarter.
(B)If an employee has alleged wrongdoing by an officer or director of the developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(e)
(g)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(f)
(h)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(g)
(i)(1)In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(h)
(j)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k)(1)This section does not impair or limit the applicability of Section 1102.5.
(2)The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following:
(A)Representatives of the University of California and other public and private academic research institutions and national laboratories.
(B)Representatives of impacted workforce labor organizations.
(C)Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
(D)Experts in technology and artificial intelligence to provide technical assistance.
(E)Personnel from other relevant departments and agencies, as necessary.
(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.
(h)The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i)The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(h)(j)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(i)(k)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 2.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Critical Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Critical risk means a foreseeable and material risk that a developers development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property, through any of the following:
(1)The creation and release of a chemical, biological, radiological, or nuclear weapon.
(2)A cyberattack.
(3)A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
(4)A foundation model evading the control of its developer or user.
(c)Developer means a person that has trained at least one foundation model with a quantity of computational power that costs at least one hundred million dollars ($100,000,000) when measured using prevailing market prices of cloud compute.
(d)Employee means a person who performs services for wages or a salary under a contract of employment, express or implied, for an employer, including both of the following:
(1)(A)Contractors or subcontractors and unpaid advisors involved with assessing, managing, or addressing the risk of critical harm from covered models and covered model derivatives.
(B)As used in this paragraph, contractor or subcontractor means a firm, corporation, partnership, or association and its responsible managing officer, as well as any supervisors, managers, and officers found by the Labor Commissioner to be personally and substantially responsible for the willful violation of this chapter.
(2)Corporate officers.
(e)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on broad data.
(2)Uses self-supervision in the training process.
(3)Applicable across a wide range of contexts.
1107.1.(a)A developer shall not make, adopt, or enforce a rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1)The developers activities pose a critical risk.
(2)The developer has made false or misleading statements about its management of critical risk.
(b)An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(c)A developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each employee of the employees rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(d)(1)A developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developers activities present a critical risk, including a monthly update to the person who made the disclosure regarding the status of the developers investigation of the disclosure and the actions taken by the developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the developer at least once each quarter.
(B)If an employee has alleged wrongdoing by an officer or director of the developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(e)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(f)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(g)(1)In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(h)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(a)There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following:
(A)Representatives of the University of California and other public and private academic research institutions and national laboratories.
(B)Representatives of impacted workforce labor organizations.
(C)Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
(D)Experts in technology and artificial intelligence to provide technical assistance.
(E)Personnel from other relevant departments and agencies, as necessary.
(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.
(h)If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(i)This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 2.Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:5.1.Whistleblower Protections: Critical Risks in AI Foundation Models1107.For purposes of this chapter:
(a)Artificial intelligence model means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Critical risk means a foreseeable and material risk that a developers development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property, through any of the following:
(1)The creation and release of a chemical, biological, radiological, or nuclear weapon.
(2)A cyberattack.
(3)A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
(4)A foundation model evading the control of its developer or user.
(c)Developer means a person that has trained at least one foundation model with a quantity of computational power that costs at least one hundred million dollars ($100,000,000) when measured using prevailing market prices of cloud compute.
(d)Employee means a person who performs services for wages or a salary under a contract of employment, express or implied, for an employer, including both of the following:
(1)(A)Contractors or subcontractors and unpaid advisors involved with assessing, managing, or addressing the risk of critical harm from covered models and covered model derivatives.
(B)As used in this paragraph, contractor or subcontractor means a firm, corporation, partnership, or association and its responsible managing officer, as well as any supervisors, managers, and officers found by the Labor Commissioner to be personally and substantially responsible for the willful violation of this chapter.
(2)Corporate officers.
(e)Foundation model means an artificial intelligence model that is all of the following:
(1)Trained on broad data.
(2)Uses self-supervision in the training process.
(3)Applicable across a wide range of contexts.
1107.1.(a)A developer shall not make, adopt, or enforce a rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1)The developers activities pose a critical risk.
(2)The developer has made false or misleading statements about its management of critical risk.
(b)An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(c)A developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1)At all times posting and displaying within any workplace maintained by the developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2)At least once each year, providing written notice to each employee of the employees rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(d)(1)A developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developers activities present a critical risk, including a monthly update to the person who made the disclosure regarding the status of the developers investigation of the disclosure and the actions taken by the developer in response to the disclosure.
(2)(A)Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the developer at least once each quarter.
(B)If an employee has alleged wrongdoing by an officer or director of the developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(e)The court is authorized to award reasonable attorneys fees to a plaintiff who brings a successful action for a violation of this section.
(f)In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(g)(1)In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2)Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3)In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4)Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5)An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(h)Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
SECTION 1.It is the intent of the Legislature to enact legislation that would establish safeguards for the development of artificial intelligence (AI) frontier models and that would build state capacity for the use of AI, that may include, but is not limited to, the findings of the Joint California Policy Working Group on AI Frontier Models, established by the Governor.It is the intent of the Legislature to enact legislation that would establish safeguards for the development of artificial intelligence (AI) frontier models and that would build state capacity for the use of AI, that may include, but is not limited to, the findings of the Joint California Policy Working Group on AI Frontier Models, established by the Governor.
|
Date
|
Status
|
|---|---|
| Sep 29, 2025 | Chaptered by Secretary of State. Chapter 138, Statutes of 2025. |
| Sep 29, 2025 | Approved by the Governor. |
| Sep 23, 2025 | Enrolled and presented to the Governor at 2 p.m. |
| Sep 13, 2025 | Assembly amendments concurred in. (Ayes 29. Noes 8. Page 3039.) Ordered to engrossing and enrolling. |
| Sep 13, 2025 | In Senate. Concurrence in Assembly amendments pending. |
| Sep 12, 2025 | Read third time. Passed. (Ayes 59. Noes 7. Page 3432.) Ordered to the Senate. |
| Sep 12, 2025 | Assembly Rule 63 suspended. |
| Sep 12, 2025 | Read second time. Ordered to third reading. |
| Sep 11, 2025 | From committee: Do pass. (Ayes 12. Noes 1.) (September 11). |
| Sep 9, 2025 | Joint Rule 62(a) suspended. |
| Sep 8, 2025 | Re-referred to Com. on P. & C.P. pursuant to Assembly Rule 77.2. |
| Sep 5, 2025 | Ordered to third reading. |
| Sep 5, 2025 | Read third time and amended. |
| Sep 5, 2025 | Assembly Rule 69(b)(1) suspended. |
| Sep 3, 2025 | Read second time. Ordered to third reading. |
| Sep 2, 2025 | Read second time and amended. Ordered to second reading. |
| Aug 29, 2025 | From committee: Do pass as amended. (Ayes 11. Noes 1.) (August 29). |
| Aug 20, 2025 | August 20 set for first hearing. Placed on APPR. suspense file. |
| Jul 17, 2025 | Read second time and amended. Re-referred to Com. on APPR. |
| Jul 16, 2025 | From committee: Do pass as amended and re-refer to Com. on APPR. (Ayes 9. Noes 0.) (July 16). |
| Jul 8, 2025 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on P. & C.P. |
| Jul 2, 2025 | From committee: Do pass and re-refer to Com. on P. & C.P. (Ayes 12. Noes 0.) (July 1). Re-referred to Com. on P. & C.P. |
| Jun 12, 2025 | Re-referred to Coms. on JUD. and P. & C.P. pursuant to Assembly Rule 96. |
| Jun 5, 2025 | Referred to Coms. on P. & C.P. and JUD. |
| May 28, 2025 | In Assembly. Read first time. Held at Desk. |
| May 28, 2025 | Read third time. Passed. (Ayes 37. Noes 0. Page 1281.) Ordered to the Assembly. |
| May 27, 2025 | Read second time. Ordered to third reading. |
| May 23, 2025 | Read second time and amended. Ordered to second reading. |
| May 23, 2025 | From committee: Do pass as amended. (Ayes 6. Noes 0. Page 1189.) (May 23). |
| May 16, 2025 | Set for hearing May 23. |
| Apr 21, 2025 | April 21 hearing: Placed on APPR. suspense file. |
| Apr 10, 2025 | Set for hearing April 21. |
| Apr 9, 2025 | From committee: Do pass and re-refer to Com. on APPR. with recommendation: To consent calendar. (Ayes 13. Noes 0. Page 705.) (April 8). Re-referred to Com. on APPR. |
| Apr 1, 2025 | Set for hearing April 8. |
| Mar 27, 2025 | Read second time and amended. Re-referred to Com. on JUD. |
| Mar 26, 2025 | From committee: Do pass as amended and re-refer to Com. on JUD. (Ayes 13. Noes 0. Page 530.) (March 25). |
| Mar 17, 2025 | Set for hearing March 25. |
| Mar 12, 2025 | Re-referred to Coms. on G.O. and JUD. |
| Feb 27, 2025 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on RLS. |
| Jan 29, 2025 | Referred to Com. on RLS. |
| Jan 8, 2025 | From printer. May be acted upon on or after February 7. |
| Jan 7, 2025 | Introduced. Read first time. To Com. on RLS. for assignment. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Sep 13, 2025 | Unfinished Business SB53 Wiener et al. Concurrence | Senate Floor |
Passed |
|
| Sep 12, 2025 | Unfinished Business SB53 Wiener et al. Concurrence | Senate Floor |
Passed |
|
| Sep 12, 2025 | SB 53 Wiener Senate Third Reading By BAUER-KAHAN | Assembly Floor |
Passed |
|
| Sep 11, 2025 | Do pass. | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
|
| Aug 29, 2025 | Do pass as amended. | Assembly Standing Committee on Appropriations |
Passed |
|
| Jul 16, 2025 | Do pass as amended and be re-referred to the Committee on [Appropriations] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
|
| Jul 1, 2025 | Do pass and be re-referred to the Committee on [Privacy and Consumer Protection] | Assembly Standing Committee on Judiciary |
Passed |
|
| May 28, 2025 | Senate 3rd Reading SB53 Wiener | Senate Floor |
Passed |
|
| May 23, 2025 | Do pass as amended | Senate Standing Committee on Appropriations |
Passed |
|
| Apr 21, 2025 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Apr 8, 2025 | Do pass, but first be re-referred to the Committee on [Appropriations] with the recommendation: To Consent Calendar | Senate Standing Committee on Judiciary |
Passed |
|
| Mar 25, 2025 | Do pass as amended, but first amend, and re-refer to the Committee on [Judiciary] | Senate Standing Committee on Governmental Organization |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| Ai for Animals | ** Support | |
| AI Futures Project | ** Support | |
| AI Lab Watch | ** Support | |
| AI Policy Tracker | ** Support | |
| All Girls Allowed | ** Support | |
| Anthropic | ** Support | |
| Anthropic Pbc | ** Support | |
| Apart Research | ** Support | |
| Association for Long Term Existence and Resilience | ** Support | |
| Association for Long Term Existence and Resilience (ALTER) | ** Support | |
| Beri | ** Support | |
| Berkeley Existential Risk Initiative | ** Support | |
| Berkeley Existential Risk Initiative (BERI) | ** Support | |
| Business Software Alliance | ** Against | |
| Business Software Association | ** Against | |
| California Chamber Of Commerce | ** Against | |
| California Democratic Party | ** Support | |
| California Federation of Labor Unions, Afl-cio | ** Support | |
| California Initiative for Technology & Democracy | ** Support | |
| California Report on Frontier AI Policy | ** Support | |
| Center for AI and Digital Policy | ** Support | |
| Center for AI Policy | ** Support | |
| center for digital democracy | ** Support | |
| Center for Human-Compatible AI | ** Support | |
| Center for Human-Compatible Artificial Intelligence, UC Berkeley | ** Support | |
| Center for Youth and Ai | ** Support | |
| Chamber Of Progress | ** Against | |
| Children's Advocacy Institute, University of San Diego School of Law | ** Support | |
| Children’s Advocacy Institute, University of San Diego School of Law | ** Support | |
| Common Sense Media | ** Support | |
| Compute Exchange | ** Support | |
| Computer & Communications Industry Association | ** Against | |
| Consumer Technology Association | ** Against | |
| Depict.ai | ** Support | |
| Design It for US | ** Support | |
| District Council Of Iron Workers Of The State Of California And Vicinity | ** Support | |
| EarningsStream LLC | ** Support | |
| Economic Security California Action (co-source) | ** Support | |
| Elicit | ** Support | |
| Encode | ** Support | |
| Encode Ai Corporation | ** Support | |
| Encode (co-source) | ** Support | |
| Encode Justice | ** Support | |
| Eon Systems | ** Support | |
| Existential Risk Observatory | ** Support | |
| Frontlines Foundation | ** Support | |
| Future Of Life Institute | ** Support | |
| Indivisible California StateStrong | ** Support | |
| Innovateedu | ** Support | |
| Insights Association | ** Against | |
| Little Hoover Commission | ** Support | |
| Los Angeles County Business Federation | ** Against | |
| Los Angeles County Business Federation (BIZFED) (UNREG) | ** Against | |
| Momentum | ** Support | |
| Mothers Against Media Addiction | ** Support | |
| Nonlinear | ** Support | |
| Noso November | ** Support | |
| Oakland Privacy | ** Support | |
| Omidyar Network | ** Support | |
| Parents Television and Media Council | ** Support | |
| Parents Together Action | ** Support | |
| Porn Free Colorado | ** Support | |
| Public Interest Privacy Center | ** Support | |
| Redwood Research | ** Support | |
| Rights4Girls | ** Support | |
| Safe AI Future | ** Support | |
| Scorecard | ** Support | |
| Secure AI Future | ** Support | |
| Secure AI Project (co-source) | ** Support | |
| Seiu Califonia | ** Support | |
| Silicon Valley Leadership Group | ** Against | |
| Techequity Action | ** Support | |
| Technet | ** Against | |
| Tech Oversight California | ** Support | |
| The Brandes Lab At Nyu | ** Support | |
| The Midas Project | ** Support | |
| The Signals Network | ** Support | |
| Transparency Coalition.ai | ** Support | |
| Trevi Digital Assets Fund | ** Support | |
| University of California | ** Support | |
| Young People's Alliance | ** Support | |
| Youth Leadership Institute | ** Support | |
| Youth Power Project | ** Support |
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 53 |
UNFINISHED BUSINESS
Bill No: SB 53
Author: Wiener (D), et al.
Amended: 9/29/25
Vote: 21
SENATE GOVERNMENTAL ORG. COMMITTEE: 13-0, 3/25/25
AYES: Padilla, Valladares, Archuleta, Ashby, Blakespear, Cervantes, Hurtado, Jones, Ochoa Bogh, Richardson, Rubio, Wahab, Weber Pierson
NO VOTE RECORDED: Dahle, Smallwood-Cuevas
SENATE JUDICIARY COMMITTEE: 13-0, 4/8/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SENATE FLOOR: 37-0, 5/28/25
AYES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
NO VOTE RECORDED: Cervantes, Limón, Reyes
ASSEMBLY FLOOR: 59-7, 9/12/25 - See last page for vote
SUBJECT: Artificial intelligence models: large developers
SOURCE: Economic Security California Action
Encode
Secure AI Project
DIGEST:
This bill requires large artificial intelligence (AI) developers, as defined, to publish safety frameworks, disclose specified transparency reports, and report critical safety incidents to the Office of Emergency Services (OES), as specified. Additionally, this bill creates enhanced whistleblower protections for employees reporting AI safety violations and establishes a consortium to design a framework for “CalCompute,” a public cloud platform to expand safe and equitable AI research, as specified.
Assembly Amendments
of 9/5/25 enact the Transparency in Frontier AI Act (TFAIA) and require large AI frontier model developers, as defined, to publish safety frameworks, disclose specified transparency reports, and report critical safety incidents to OES, as specified.
ANALYSIS:
Existing law:
- Generally regulates AI, including by requiring that a generative AI system or service made publicly available to Californians to use, the developer of the system or service to post on their internet website documentation regarding the data used by the developer to train the AI system or service, as specified.
- Establishes the California Department of Technology (CDT), within the Government Operations Agency (GovOps), as specified.
- Establishes the California Cybersecurity Integration Center, within OES, to serve as the central organizing hub of state government’s cybersecurity activities and to coordinate information sharing with various entities.
This bill:
- Requires a large AI frontier developer, as defined, to write, implement, and comply with, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developer’s frontier models and describes how the large frontier developer approaches specified related situations.
- Requires a frontier developer, before deploying a new frontier model or a substantially modified version of an existing frontier model, to clearly and conspicuously publish on its internet website a transparency report, as specified.
- Requires a large frontier developer to transmit to OES a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months or pursuant to another reasonable schedule, as appropriate.
- Requires OES to establish a mechanism to be used by a frontier developer or a member of the public to report a critical safety incident, as specified.
- Requires OES, beginning January 1, 2027, and annually thereafter, to produce a report with anonymized and aggregated information about critical safety incidents that have been reviewed by the office since the preceding report, as specified.
- Requires CDT, on or before January 1, 2027, and annually thereafter, to assess recent evidence and developments relevant to the purposes of this bill and to make recommendations about whether and how to update specified and related definitions in statute.
- Provides that if a large frontier developer fails to publish or transmit a complaint document as required by this bill, makes a statement in violation of this bill, or fails to comply with its own AI framework shall be subject to a civil penalty in an amount dependent upon the severity of the violation that does not exceed one million dollars ($1,000,000) per violation.
- Establishes a consortium, within GovOps to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute,” as specified.
- Prohibits a frontier developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents a covered employee – as defined – from disclosing, or retaliating against a covered employee for disclosing, information to the Attorney General (AG), a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, as specified.
Background
Author Statement. According to the author’s office, “in 2024, as part of his veto of Senate Bill 1047 (Wiener), Governor Newsom’s Joint California Working Group on AI Frontier Models was established – a group of top experts tasked with charting a course forward on AI policy for the developers of the most advanced AI systems. Their final report, released in June 2025, emphasized the growing evidence for risk of severe harm, such as ‘AI-enabled hacking or biological attacks, and loss of control’ and argued ‘California has a unique opportunity to continue supporting developments in frontier AI while addressing substantial risks that could have far-reaching consequences for the state and beyond.
“Drawing recommendations from Governor Newsom’s working group report, Senate Bill 53 requires covered developers to write, implement, and publish their safety and security protocol in redacted form to protect intellectual property. It would also require covered developers to report certain, carefully defined critical safety incidents to the Attorney General and would allow members of the public to report incidents.
“SB 53 only applies to AI companies that have trained a model with 1026 floating point operations (FLOPs), a measure of computational power. These companies are spending hundreds of millions of dollars to train the most advanced AI models. As recommended by the Report, SB 53 also authorizes the Attorney General to adjust the scoping of the bill in the future to keep up with technological developments, but only focuses on well-resourced AI companies at the frontier of AI development.
“Senate Bill 53 strengthens whistleblower protections for employees of frontier artificial intelligence laboratory companies whose activities pose a catastrophic risk. SB 53 also establishes a consortium to help create CalCompute: a public AI research cluster that will provide startups and researchers with access to the resources needed to develop large-scale AI systems.”
Frontier AI Models. “Foundation” or “frontier” models are the largest, most powerful AI systems being built today. Because of their broad capabilities, they have the potential to unlock major breakthroughs in science and medicine, streamline complex processes, and grow the economy. However, they also carry the potential for catastrophic risks. Critics point out that a frontier model might be used to cure disease or, conversely, to engineer a new pandemic. It could automate government functions or, if left unchecked, disrupt critical infrastructure.
Debates around AI safety often center on timing: some argue that regulating too early risks stifling innovation, while others warn that waiting until evidence of harm is undeniable may leave society unable to respond. In 2024, SB 1047 (Wiener) attempted to regulate frontier models directly. It would have required developers to adopt safety and security protocols before training, install shutdown mechanisms, conduct risk assessments, and submit to third-party audits. It also would have barred release of models posing unreasonable catastrophic risks and created a new Board of Frontier Models to oversee compliance. Governor Newsom vetoed the bill, acknowledging the urgency but stressing that any regulatory framework must be grounded in empirical evidence and able to evolve alongside rapidly advancing technology.
Following the veto, the Governor convened the Joint California Policy Working Group on AI Frontier Models, which issued a final policy report in June 2025. SB 53 attempts to implement some of those recommendations with a narrower, transparency-focused approach. Rather than dictating safety standards, the bill requires the largest frontier developers (those training models with extraordinary compute power and earning over $500 million in annual revenue) to publish both safety protocols and detailed transparency reports for each model. Developers that meet only the compute threshold must release a more limited transparency report. These documents must explain whether and how catastrophic risks (defined as events causing more than 50 deaths or $1 billion in damage) are assessed and mitigated.
SB 53 also sets up a critical incident reporting system through OES, requiring developers to report incidents within 15 days, or within 24 hours if there is an imminent threat. CDT is empowered to recommend updates to the law as AI capabilities evolve.
CalCompute. This bill also creates a consortium to establish a framework for creating “CalCompute,” a public cloud cluster within GovOps to expand access to compute for AI research and safety testing, and extends whistleblower protections to employees who report risks or violations. Cluster computing is a type of computing where multiple computers are connected to work together as a single system. Computing clusters typically consist of servers, workstations, and personal computers that communicate over a local area network or a wide area network.
Whistleblower Protections. This bill adds safeguards from the Working Group Report: capability thresholds must be disclosed in protocols and reports, mitigation steps must be documented when thresholds are exceeded, and OES and the Attorney General must issue annual, anonymized summaries of critical incidents and whistleblower disclosures.
Whistleblower protections within companies developing AI foundation models can be crucial given the extraordinary scale and impact of the technology. These protections serve as an essential safety valve in an industry where the potential consequences of unethical or dangerous development practices can affect millions, if not billions, of people.
These models increasingly power critical infrastructure across healthcare, finance, employment, education, and government services. When employees with direct knowledge of risks, harms, or unethical practices cannot safely speak up, dangerous systems may be deployed without proper safeguards or public awareness. Whistleblowers often represent the last line of defense when corporate incentives prioritize growth, profit, or competitive advantage over public welfare.
Related/Prior Legislation
SB 1047 (Wiener, 2024) would have required developers of powerful artificial intelligence models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. This bill would have also established a state entity to oversee the development of these models and called for the creation of a consortium to develop a framework for a public cloud computing cluster. (Vetoed by Governor Newsom)
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Assembly Appropriations Committee, costs (General Fund) to the Department of Justice (DOJ), likely in the low millions of dollars annually, to establish reporting mechanisms, review critical incident reports, conduct investigations, publish reports, and enforce violations. DOJ anticipates cots of approximately $1.1 million in fiscal year 2025-26 and $2 million annually ongoing thereafter for eight staff positions (attorneys, analysts, IT specialists, and legal secretaries) in its Consumer Protection Section and external consultant costs. DOJ reports it is unable to absorb these costs and can implement this bill only with an appropriation of additional funding. DOJ may also incur enforcement costs for violations of the bill′s whistleblower protections; the bill does not clearly specify the entity responsible for this enforcement but permits an employee making a whistleblower report to use an existing DOJ whistleblower hotline.
Costs (General Fund) to GovOps to establish and operate the CalCompute consortium until January 1, 2027, possibly in the high hundreds of thousands of dollars to low millions of dollars. GovOps estimates total costs of $2.5 million for expert contractors, infrastructure planning, and staffing to manage the project, conduct research, and develop the required report. GovOps was not able to provide a breakdown of these costs but anticipates the workload would be handled by contract workers due to the short timeframe in the bill. Members of the consortium are not entitled to compensation but are entitled to reimbursement for necessary expenses incurred in performing their duties; these costs were not included in GovOps′ fiscal estimate but may be in the thousands to low tens of thousands of dollars depending on the activities of the consortium.
Possible cost pressures (General Fund) of an unknown but potentially significant amount to the University of California (UC) to operate CalCompute should it be established within the UC. State costs may be offset to some extent by private donations, which the bill authorizes the UC to receive to implement CalCompute.
Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce violations of the bill′s whistleblower provisions. Actual costs will depend on the number of violations, the number of actions pursued, and the amount of workload associated with each action.
Cost pressures (Trial Court Trust Fund, General Fund) to the courts to adjudicate enforcement actions and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2025-26 state budget provides $82 million ongoing General Fund to the Trial Court Trust Fund for court operations.
SUPPORT: (Verified 9/9/2025)
Economic Security California Action (co-source)
Encode (co-source)
Secure AI Project (co-source)
AI for Animals
AI Futures Project
AI Lab Watch
AI Policy Tracker
All Girls Allowed
Apart Research
Association for Long Term Existence and Resilience
Berkeley Existential Risk Initiative
California Democratic Party
California Federation of Labor Unions, AFL-CIO
California Initiative for Technology & Democracy
Center for AI and Digital Policy
Center for AI Policy
Center for Human-Compatible AI
Center for Youth and AI
Children's Advocacy Institute, University of San Diego School of Law
Common Sense Media
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
EarningsStream LLC
Elicit
Eon Systems
Existential Risk Observatory
Frontlines Foundation
Future of Life Institute
Indivisible California Statestrong
Little Hoover Commission
Momentum
Nonlinear
Oakland Privacy
Omidyar Network
Redwood Research
Safe AI Future
Scorecard
Secure Ai Future
SEIU Califonia
Tech Oversight California
TechEquity Action
The Brandes Lab At NYU
The Midas Project
The Signals Network
Transparency Coalition.ai
Trevi Digital Assets Fund
University of California
Youth Leadership Institute
OPPOSITION: (Verified 9/9/25)
Business Software Alliance
California Chamber of Commerce
Computer & Communications Industry Association
Consumer Technology Association
Insights Association
Los Angeles County Business Federation
Silicon Valley Leadership Group
TechNet
ARGUMENTS IN SUPPORT: In support of the bill, Secure AI Project, co-sponsors of the bill, alongside a coalition of technology equity advocacy groups write that:
The California Report on Frontier AI Policy, while it does not endorse any specific legislation, forms the foundation for SB 53. Established by Governor Newsom in 2024 and led by Dr. Fei-Fei Li, Dr. Jennifer Tour Chayes and Mariano-Florentino Cuéllar, the report is anchored on the notion of ‘trust but verify’ and calls for more transparency into the safety practices of AI companies, adverse event reporting requirements, and whistleblower protections. SB 53 implements these principles.
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The Report stated that ‘some risks have unclear but growing evidence...AI-enabled hacking or biological attacks, and loss of control’ – the risks that SB 53 aims to address and gather more evidence about. Advanced AI is currently mostly unregulated, and these risks are currently being managed by companies themselves without any requirement that they inform the public about their risk management practices or report serious incidents. SB 53 addresses this much needed gap by implementing four key recommendations from the report.
First, the Report argued that ‘transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk.’ SB 53 implements this recommendation as a requirement for large AI developers to write, publish, and follow safety and security protocols to manage the most severe risks. This is in line with voluntary commitments that companies have already made. Rather than prescribe specific technical standards that companies must take, the bill simply requires companies to be transparent about the approaches they are using. Some of the specific required elements of safety protocols, such as a requirement to manage risks related to internal use of AI models and cybersecurity policies, directly mirror recommendations in the Report. Others mirror components of the Stanford Foundation Model Transparency Index, which is cited prominently in the Report.
Second, the Report stated that ‘transparency into pre-deployment assessments of capabilities and risks, spanning both developer-conducted and externally conducted evaluations, is vital given that these evaluations are early indicators of how models may affect society and may be interpreted (potentially undesirably) as safety assurances.’ SB 53 accomplishes this with a requirement that large developers publish transparency reports that include the results of their pre-deployment assessments of catastrophic risk. The Report also argues that ‘transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety,’ which forms the basis for 22757.12(c)(3).
Third, the Report concluded that ‘an adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base.’ SB 53 takes exactly this approach by establishing a tightly defined set of critical safety incidents that AI developers are required to report to the Attorney General. It would also allow members of the public to optionally submit reports.
Finally, the Report recommends strengthening whistleblower protections, pointing out that ‘actions that may clearly pose a risk and violate company policies...may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of activities, which may draw upon notions of ′good faith′ reporting on risks found in other domains such as cybersecurity.’ This recommendation is mirrored in SB 53, which allows employees to report evidence of catastrophic risks as well as violations of SB 53 itself to government authorities with legal protections against retaliation.
SB 53 only applies to the largest AI developers – those training models with more than 10^26 floating point operations (FLOPs). These are companies spending hundreds of millions or billions of dollars to train the most advanced AI models. It would impose no burden on smaller companies and the requirements it imposes on large companies are minimal compared to what companies are already voluntarily doing. The Report argues that ‘policymakers should ensure that mechanisms are in place to adapt thresholds over time—not only by updating specific threshold values but also by revising or replacing metrics if needed.’ It also suggests specific criteria that thresholds should be evaluated for. Following this recommendation, SB 53 allows the Attorney General to update the definition of ‘large developer’ through regulation while considering the same factors described in the report. Regardless of any update, the Attorney General must only include ‘well-resourced large developers at the frontier of artificial intelligence development’ in the scoping of the bill. If legislation is needed to cover other developers, the Attorney General is instructed to write a report to the Legislature requesting it.
Finally, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California′s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. This mirrors a similar computing cluster that is already being established in New York state. We support this groundbreaking effort, which would advance and democratize AI research in California.
SB 53 thoughtfully implements the recommendations of the Report by combining a low-burden transparency and reporting regime with a public compute cluster that will broaden access for AI researchers and startups in California. This is a commonsense approach that will strengthen the AI ecosystem, benefiting both companies and the public interest.”
ARGUMENTS IN OPPOSITION: In opposition to this bill, the Chamber of Progress argues:
On behalf of the Chamber of Progress, a tech industry association supporting public policies to build a more inclusive society in which all people benefit from technological advances, we respectfully urge you to oppose SB 53, based on its recent amendments.
The definition of ″catastrophic risk″ remains vague and overreaching
While the amended bill replaces the term ″critical risk″ with ″catastrophic risk,″ the underlying problem persists. The definition remains overly expansive and ambiguous, capturing a wide array of hypothetical scenarios that may not reflect real-world AI capabilities or threats.
Under Section 22757.11(b), the definition of ″catastrophic risk″ includes scenarios where a foundation model is ″materially likely″ to cause harm, potentially due to misuse or malicious inputs. However, this standard is vague, lacks clear and objective thresholds, and leaves room for subjective interpretation by whistleblowers or regulators. In a rapidly evolving field like AI, such ambiguity could unfairly penalize developers who are acting responsibly.
In addition, the inclusion of highly abstract risks, such as the evasion of human control under Section 22757.12(a)(2), creates significant uncertainty. Without clear technical criteria, companies may face liability or investigation based on assumptions about what a model might enable rather than what it has demonstrably done. This uncertainty undermines research and commercial deployment in California and could push critical AI development efforts out of state or abroad.
The $100,000,000 compute cost threshold risks misidentifying frontier AI models
SB 53′s use of an arbitrary $100,000,000 compute cost threshold to determine eligibility for protections is an inherently flawed method for identifying frontier AI models. This threshold may result in the overinclusion of developers working on benign systems while potentially excluding smaller models that pose significant real-world risks. It also ignores the constantly changing cost of compute.
A more effective approach would involve a threshold based on model capabilities, deployment context, and specific use cases rather than relying solely on computational costs.
SB 53′s extensive safety and security protocols create impractical burdens for AI developers
SB 53 imposes comprehensive safety and security requirements on AI developers, as outlined in Section 22757.12(a), including risk testing, deployment practices, and escalation procedures. While these objectives are important, the bill demands an impractical level of detailed planning and documentation for every conceivable misuse scenario, many of which are speculative or unrealistic.
This exhaustive approach compels developers to allocate significant time and resources toward preparing for hypothetical risks rather than addressing actual, demonstrable harms. For startups and smaller companies, these extensive protocols create a heavy administrative burden that diverts critical resources away from innovation and the timely deployment of beneficial AI technologies.
Additionally, Section 22757.12(c)′s requirement that developers publish detailed transparency reports, before or at the time of deploying a new or substantially modified foundation model, creates significant risks to both competitiveness and operational security.
Although redactions are permitted under subsection (f), the requirement to publish the ″character and justification″ of redacted material could still inadvertently expose business-sensitive strategies or vulnerabilities. This level of forced transparency goes beyond reasonable accountability and may discourage responsible companies from operating in California. It also creates opportunities for misuse by malicious actors who could exploit disclosed model weaknesses or mitigation gaps.
In fast-moving AI markets, publication of this level of detail erodes a developer′s ability to maintain a competitive edge and deters innovation by raising legal and reputational risks associated with even speculative harms.
ASSEMBLY FLOOR: 59-7, 9/12/25
AYES: Addis, Aguiar-Curry, Ahrens, Alvarez, Arambula, Ávila Farías, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Caloza, Carrillo, Connolly, Dixon, Elhawary, Fong, Garcia, Gipson, Jeff Gonzalez, Mark González, Haney, Harabedian, Hart, Irwin, Johnson, Kalra, Krell, Lee, Lowenthal, Muratsuchi, Nguyen, Ortega, Pacheco, Papan, Patel, Pellerin, Petrie-Norris, Quirk-Silva, Ransom, Celeste Rodriguez, Michelle Rodriguez, Rogers, Blanca Rubio, Sanchez, Schiavo, Schultz, Sharp-Collins, Solache, Soria, Stefani, Valencia, Ward, Wicks, Wilson, Zbur, Rivas
NOES: DeMaio, Ellis, Hadwick, Hoover, Macedo, Tangipa, Wallis
NO VOTE RECORDED: Alanis, Bains, Castillo, Chen, Davies, Flora, Gabriel, Gallagher, Jackson, Lackey, McKinnor, Patterson, Ramos, Ta
Prepared by: Brian Duke / G.O. / (916) 651-1530
7/21/26 11:03:04
**** END ****
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 53 |
UNFINISHED BUSINESS
Bill No: SB 53
Author: Wiener (D), et al.
Amended: 9/5/25 in Assembly
Vote: 21
SENATE GOVERNMENTAL ORG. COMMITTEE: 13-0, 3/25/25
AYES: Padilla, Valladares, Archuleta, Ashby, Blakespear, Cervantes, Hurtado, Jones, Ochoa Bogh, Richardson, Rubio, Wahab, Weber Pierson
NO VOTE RECORDED: Dahle, Smallwood-Cuevas
SENATE JUDICIARY COMMITTEE: 13-0, 4/8/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SENATE FLOOR: 37-0, 5/28/25
AYES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
NO VOTE RECORDED: Cervantes, Limón, Reyes
ASSEMBLY FLOOR: 45-7, 9/12/25 – Roll call vote not available.
SUBJECT: Artificial intelligence models: large developers
SOURCE: Economic Security California Action
Encode
Secure AI Project
DIGEST:
This bill requires large artificial intelligence (AI) developers, as defined, to publish safety frameworks, disclose specified transparency reports, and report critical safety incidents to the Office of Emergency Services (OES), as specified. Additionally, this bill creates enhanced whistleblower protections for employees reporting AI safety violations and establishes a consortium to design a framework for “CalCompute,” a public cloud platform to expand safe and equitable AI research, as specified.
Assembly Amendments
of 9/5/25 enact the Transparency in Frontier AI Act (TFAIA) and require large AI frontier model developers, as defined, to publish safety frameworks, disclose specified transparency reports, and report critical safety incidents to OES, as specified.
ANALYSIS:
Existing law:
- Generally regulates AI, including by requiring that a generative AI system or service made publicly available to Californians to use, the developer of the system or service to post on their internet website documentation regarding the data used by the developer to train the AI system or service, as specified.
- Establishes the California Department of Technology (CDT), within the Government Operations Agency (GovOps), as specified.
- Establishes the California Cybersecurity Integration Center, within OES, to serve as the central organizing hub of state government’s cybersecurity activities and to coordinate information sharing with various entities.
This bill:
- Requires a large AI frontier developer, as defined, to write, implement, and comply with, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developer’s frontier models and describes how the large frontier developer approaches specified related situations.
- Requires a frontier developer, before deploying a new frontier model or a substantially modified version of an existing frontier model, to clearly and conspicuously publish on its internet website a transparency report, as specified.
- Requires a large frontier developer to transmit to OES a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months or pursuant to another reasonable schedule, as appropriate.
- Requires OES to establish a mechanism to be used by a frontier developer or a member of the public to report a critical safety incident, as specified.
- Requires OES, beginning January 1, 2027, and annually thereafter, to produce a report with anonymized and aggregated information about critical safety incidents that have been reviewed by the office since the preceding report, as specified.
- Requires CDT, on or before January 1, 2027, and annually thereafter, to assess recent evidence and developments relevant to the purposes of this bill and to make recommendations about whether and how to update specified and related definitions in statute.
- Provides that if a large frontier developer fails to publish or transmit a complaint document as required by this bill, makes a statement in violation of this bill, or fails to comply with its own AI framework shall be subject to a civil penalty in an amount dependent upon the severity of the violation that does not exceed one million dollars ($1,000,000) per violation.
- Establishes a consortium, within GovOps to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute,” as specified.
- Prohibits a frontier developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents a covered employee – as defined – from disclosing, or retaliating against a covered employee for disclosing, information to the Attorney General (AG), a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, as specified.
Background
Author Statement. According to the author’s office, “in 2024, as part of his veto of Senate Bill 1047 (Wiener), Governor Newsom’s Joint California Working Group on AI Frontier Models was established – a group of top experts tasked with charting a course forward on AI policy for the developers of the most advanced AI systems. Their final report, released in June 2025, emphasized the growing evidence for risk of severe harm, such as ‘AI-enabled hacking or biological attacks, and loss of control’ and argued ‘California has a unique opportunity to continue supporting developments in frontier AI while addressing substantial risks that could have far-reaching consequences for the state and beyond.
“Drawing recommendations from Governor Newsom’s working group report, Senate Bill 53 requires covered developers to write, implement, and publish their safety and security protocol in redacted form to protect intellectual property. It would also require covered developers to report certain, carefully defined critical safety incidents to the Attorney General and would allow members of the public to report incidents.
“SB 53 only applies to AI companies that have trained a model with 1026 floating point operations (FLOPs), a measure of computational power. These companies are spending hundreds of millions of dollars to train the most advanced AI models. As recommended by the Report, SB 53 also authorizes the Attorney General to adjust the scoping of the bill in the future to keep up with technological developments, but only focuses on well-resourced AI companies at the frontier of AI development.
“Senate Bill 53 strengthens whistleblower protections for employees of frontier artificial intelligence laboratory companies whose activities pose a catastrophic risk. SB 53 also establishes a consortium to help create CalCompute: a public AI research cluster that will provide startups and researchers with access to the resources needed to develop large-scale AI systems.”
Frontier AI Models. “Foundation” or “frontier” models are the largest, most powerful AI systems being built today. Because of their broad capabilities, they have the potential to unlock major breakthroughs in science and medicine, streamline complex processes, and grow the economy. However, they also carry the potential for catastrophic risks. Critics point out that a frontier model might be used to cure disease or, conversely, to engineer a new pandemic. It could automate government functions or, if left unchecked, disrupt critical infrastructure.
Debates around AI safety often center on timing: some argue that regulating too early risks stifling innovation, while others warn that waiting until evidence of harm is undeniable may leave society unable to respond. In 2024, SB 1047 (Wiener) attempted to regulate frontier models directly. It would have required developers to adopt safety and security protocols before training, install shutdown mechanisms, conduct risk assessments, and submit to third-party audits. It also would have barred release of models posing unreasonable catastrophic risks and created a new Board of Frontier Models to oversee compliance. Governor Newsom vetoed the bill, acknowledging the urgency but stressing that any regulatory framework must be grounded in empirical evidence and able to evolve alongside rapidly advancing technology.
Following the veto, the Governor convened the Joint California Policy Working Group on AI Frontier Models, which issued a final policy report in June 2025. SB 53 attempts to implement some of those recommendations with a narrower, transparency-focused approach. Rather than dictating safety standards, the bill requires the largest frontier developers (those training models with extraordinary compute power and earning over $500 million in annual revenue) to publish both safety protocols and detailed transparency reports for each model. Developers that meet only the compute threshold must release a more limited transparency report. These documents must explain whether and how catastrophic risks (defined as events causing more than 50 deaths or $1 billion in damage) are assessed and mitigated.
SB 53 also sets up a critical incident reporting system through OES, requiring developers to report incidents within 15 days, or within 24 hours if there is an imminent threat. CDT is empowered to recommend updates to the law as AI capabilities evolve.
CalCompute. This bill also creates a consortium to establish a framework for creating “CalCompute,” a public cloud cluster within GovOps to expand access to compute for AI research and safety testing, and extends whistleblower protections to employees who report risks or violations. Cluster computing is a type of computing where multiple computers are connected to work together as a single system. Computing clusters typically consist of servers, workstations, and personal computers that communicate over a local area network or a wide area network.
Whistleblower Protections. This bill adds safeguards from the Working Group Report: capability thresholds must be disclosed in protocols and reports, mitigation steps must be documented when thresholds are exceeded, and OES and the Attorney General must issue annual, anonymized summaries of critical incidents and whistleblower disclosures.
Whistleblower protections within companies developing AI foundation models can be crucial given the extraordinary scale and impact of the technology. These protections serve as an essential safety valve in an industry where the potential consequences of unethical or dangerous development practices can affect millions, if not billions, of people.
These models increasingly power critical infrastructure across healthcare, finance, employment, education, and government services. When employees with direct knowledge of risks, harms, or unethical practices cannot safely speak up, dangerous systems may be deployed without proper safeguards or public awareness. Whistleblowers often represent the last line of defense when corporate incentives prioritize growth, profit, or competitive advantage over public welfare.
Related/Prior Legislation
SB 1047 (Wiener, 2024) would have required developers of powerful artificial intelligence models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. This bill would have also established a state entity to oversee the development of these models and called for the creation of a consortium to develop a framework for a public cloud computing cluster. (Vetoed by Governor Newsom)
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Assembly Appropriations Committee, costs (General Fund) to the Department of Justice (DOJ), likely in the low millions of dollars annually, to establish reporting mechanisms, review critical incident reports, conduct investigations, publish reports, and enforce violations. DOJ anticipates cots of approximately $1.1 million in fiscal year 2025-26 and $2 million annually ongoing thereafter for eight staff positions (attorneys, analysts, IT specialists, and legal secretaries) in its Consumer Protection Section and external consultant costs. DOJ reports it is unable to absorb these costs and can implement this bill only with an appropriation of additional funding. DOJ may also incur enforcement costs for violations of the bill′s whistleblower protections; the bill does not clearly specify the entity responsible for this enforcement but permits an employee making a whistleblower report to use an existing DOJ whistleblower hotline.
Costs (General Fund) to GovOps to establish and operate the CalCompute consortium until January 1, 2027, possibly in the high hundreds of thousands of dollars to low millions of dollars. GovOps estimates total costs of $2.5 million for expert contractors, infrastructure planning, and staffing to manage the project, conduct research, and develop the required report. GovOps was not able to provide a breakdown of these costs but anticipates the workload would be handled by contract workers due to the short timeframe in the bill. Members of the consortium are not entitled to compensation but are entitled to reimbursement for necessary expenses incurred in performing their duties; these costs were not included in GovOps′ fiscal estimate but may be in the thousands to low tens of thousands of dollars depending on the activities of the consortium.
Possible cost pressures (General Fund) of an unknown but potentially significant amount to the University of California (UC) to operate CalCompute should it be established within the UC. State costs may be offset to some extent by private donations, which the bill authorizes the UC to receive to implement CalCompute.
Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce violations of the bill′s whistleblower provisions. Actual costs will depend on the number of violations, the number of actions pursued, and the amount of workload associated with each action.
Cost pressures (Trial Court Trust Fund, General Fund) to the courts to adjudicate enforcement actions and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2025-26 state budget provides $82 million ongoing General Fund to the Trial Court Trust Fund for court operations.
SUPPORT: (Verified 9/9/2025)
Economic Security California Action (co-source)
Encode (co-source)
Secure AI Project (co-source)
AI for Animals
AI Futures Project
AI Lab Watch
AI Policy Tracker
All Girls Allowed
Apart Research
Association for Long Term Existence and Resilience
Berkeley Existential Risk Initiative
California Democratic Party
California Federation of Labor Unions, AFL-CIO
California Initiative for Technology & Democracy
Center for AI and Digital Policy
Center for AI Policy
Center for Human-Compatible AI
Center for Youth and AI
Children's Advocacy Institute, University of San Diego School of Law
Common Sense Media
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
EarningsStream LLC
Elicit
Eon Systems
Existential Risk Observatory
Frontlines Foundation
Future of Life Institute
Indivisible California Statestrong
Little Hoover Commission
Momentum
Nonlinear
Oakland Privacy
Omidyar Network
Redwood Research
Safe AI Future
Scorecard
Secure Ai Future
SEIU Califonia
Tech Oversight California
TechEquity Action
The Brandes Lab At NYU
The Midas Project
The Signals Network
Transparency Coalition.ai
Trevi Digital Assets Fund
University of California
Youth Leadership Institute
OPPOSITION: (Verified 9/9/25)
Business Software Alliance
California Chamber of Commerce
Computer & Communications Industry Association
Consumer Technology Association
Insights Association
Los Angeles County Business Federation
Silicon Valley Leadership Group
TechNet
ARGUMENTS IN SUPPORT: In support of the bill, Secure AI Project, co-sponsors of the bill, alongside a coalition of technology equity advocacy groups write that:
The California Report on Frontier AI Policy, while it does not endorse any specific legislation, forms the foundation for SB 53. Established by Governor Newsom in 2024 and led by Dr. Fei-Fei Li, Dr. Jennifer Tour Chayes and Mariano-Florentino Cuéllar, the report is anchored on the notion of ‘trust but verify’ and calls for more transparency into the safety practices of AI companies, adverse event reporting requirements, and whistleblower protections. SB 53 implements these principles.
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The Report stated that ‘some risks have unclear but growing evidence...AI-enabled hacking or biological attacks, and loss of control’ – the risks that SB 53 aims to address and gather more evidence about. Advanced AI is currently mostly unregulated, and these risks are currently being managed by companies themselves without any requirement that they inform the public about their risk management practices or report serious incidents. SB 53 addresses this much needed gap by implementing four key recommendations from the report.
First, the Report argued that ‘transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk.’ SB 53 implements this recommendation as a requirement for large AI developers to write, publish, and follow safety and security protocols to manage the most severe risks. This is in line with voluntary commitments that companies have already made. Rather than prescribe specific technical standards that companies must take, the bill simply requires companies to be transparent about the approaches they are using. Some of the specific required elements of safety protocols, such as a requirement to manage risks related to internal use of AI models and cybersecurity policies, directly mirror recommendations in the Report. Others mirror components of the Stanford Foundation Model Transparency Index, which is cited prominently in the Report.
Second, the Report stated that ‘transparency into pre-deployment assessments of capabilities and risks, spanning both developer-conducted and externally conducted evaluations, is vital given that these evaluations are early indicators of how models may affect society and may be interpreted (potentially undesirably) as safety assurances.’ SB 53 accomplishes this with a requirement that large developers publish transparency reports that include the results of their pre-deployment assessments of catastrophic risk. The Report also argues that ‘transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety,’ which forms the basis for 22757.12(c)(3).
Third, the Report concluded that ‘an adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base.’ SB 53 takes exactly this approach by establishing a tightly defined set of critical safety incidents that AI developers are required to report to the Attorney General. It would also allow members of the public to optionally submit reports.
Finally, the Report recommends strengthening whistleblower protections, pointing out that ‘actions that may clearly pose a risk and violate company policies...may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of activities, which may draw upon notions of ′good faith′ reporting on risks found in other domains such as cybersecurity.’ This recommendation is mirrored in SB 53, which allows employees to report evidence of catastrophic risks as well as violations of SB 53 itself to government authorities with legal protections against retaliation.
SB 53 only applies to the largest AI developers – those training models with more than 10^26 floating point operations (FLOPs). These are companies spending hundreds of millions or billions of dollars to train the most advanced AI models. It would impose no burden on smaller companies and the requirements it imposes on large companies are minimal compared to what companies are already voluntarily doing. The Report argues that ‘policymakers should ensure that mechanisms are in place to adapt thresholds over time—not only by updating specific threshold values but also by revising or replacing metrics if needed.’ It also suggests specific criteria that thresholds should be evaluated for. Following this recommendation, SB 53 allows the Attorney General to update the definition of ‘large developer’ through regulation while considering the same factors described in the report. Regardless of any update, the Attorney General must only include ‘well-resourced large developers at the frontier of artificial intelligence development’ in the scoping of the bill. If legislation is needed to cover other developers, the Attorney General is instructed to write a report to the Legislature requesting it.
Finally, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California′s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. This mirrors a similar computing cluster that is already being established in New York state. We support this groundbreaking effort, which would advance and democratize AI research in California.
SB 53 thoughtfully implements the recommendations of the Report by combining a low-burden transparency and reporting regime with a public compute cluster that will broaden access for AI researchers and startups in California. This is a commonsense approach that will strengthen the AI ecosystem, benefiting both companies and the public interest.”
ARGUMENTS IN OPPOSITION: In opposition to this bill, the Chamber of Progress argues:
On behalf of the Chamber of Progress, a tech industry association supporting public policies to build a more inclusive society in which all people benefit from technological advances, we respectfully urge you to oppose SB 53, based on its recent amendments.
The definition of ″catastrophic risk″ remains vague and overreaching
While the amended bill replaces the term ″critical risk″ with ″catastrophic risk,″ the underlying problem persists. The definition remains overly expansive and ambiguous, capturing a wide array of hypothetical scenarios that may not reflect real-world AI capabilities or threats.
Under Section 22757.11(b), the definition of ″catastrophic risk″ includes scenarios where a foundation model is ″materially likely″ to cause harm, potentially due to misuse or malicious inputs. However, this standard is vague, lacks clear and objective thresholds, and leaves room for subjective interpretation by whistleblowers or regulators. In a rapidly evolving field like AI, such ambiguity could unfairly penalize developers who are acting responsibly.
In addition, the inclusion of highly abstract risks, such as the evasion of human control under Section 22757.12(a)(2), creates significant uncertainty. Without clear technical criteria, companies may face liability or investigation based on assumptions about what a model might enable rather than what it has demonstrably done. This uncertainty undermines research and commercial deployment in California and could push critical AI development efforts out of state or abroad.
The $100,000,000 compute cost threshold risks misidentifying frontier AI models
SB 53′s use of an arbitrary $100,000,000 compute cost threshold to determine eligibility for protections is an inherently flawed method for identifying frontier AI models. This threshold may result in the overinclusion of developers working on benign systems while potentially excluding smaller models that pose significant real-world risks. It also ignores the constantly changing cost of compute.
A more effective approach would involve a threshold based on model capabilities, deployment context, and specific use cases rather than relying solely on computational costs.
SB 53′s extensive safety and security protocols create impractical burdens for AI developers
SB 53 imposes comprehensive safety and security requirements on AI developers, as outlined in Section 22757.12(a), including risk testing, deployment practices, and escalation procedures. While these objectives are important, the bill demands an impractical level of detailed planning and documentation for every conceivable misuse scenario, many of which are speculative or unrealistic.
This exhaustive approach compels developers to allocate significant time and resources toward preparing for hypothetical risks rather than addressing actual, demonstrable harms. For startups and smaller companies, these extensive protocols create a heavy administrative burden that diverts critical resources away from innovation and the timely deployment of beneficial AI technologies.
Additionally, Section 22757.12(c)′s requirement that developers publish detailed transparency reports, before or at the time of deploying a new or substantially modified foundation model, creates significant risks to both competitiveness and operational security.
Although redactions are permitted under subsection (f), the requirement to publish the ″character and justification″ of redacted material could still inadvertently expose business-sensitive strategies or vulnerabilities. This level of forced transparency goes beyond reasonable accountability and may discourage responsible companies from operating in California. It also creates opportunities for misuse by malicious actors who could exploit disclosed model weaknesses or mitigation gaps.
In fast-moving AI markets, publication of this level of detail erodes a developer′s ability to maintain a competitive edge and deters innovation by raising legal and reputational risks associated with even speculative harms.
Prepared by: Brian Duke / G.O. / (916) 651-1530
9/12/25 22:23:23
**** END ****
SENATE THIRD READING
STR Bill Id:SB 53
Author:(Wiener)
As Amended Ver:September 5, 2025
Majority vote
SUMMARY
Requires developers of the most advanced, costly artificial intelligence (AI) systems to implement certain protocols and publically disclose the protocols they use to mitigate the risk of catastrophic harms. Requires the Office of Emergency Management to establish a mechanism by which developers and the public can report critical safety incidents. Provides for whistleblower protections and enforcement by the Attorney General (AG). Requires the Department of Technology to offer guidance to the Legislature on refining the scope over time to reflect technological advances. Provides, upon appropriation, for the creation of a framework to create a public cloud computing cluster.
Major Provisions
1) Requires large frontier developer who train AI models on a specified amount of compute and with revenues above $500 million to write, implement, comply with, and clearly and conspicuously publish on its internet website a frontier AI framework which details how the developer will address catastrophic harms and the protocols in place to address the materialization of such harms.
2) Requires a large frontier developer, before or at the time of making a new foundation model available, to publish on their internet website a transparency report for the model that describes the risk assessments or risk mitigation assessments used by the developer during the development of the model, including whether the developer used third-parties in the assessments and whether any denoted risk thresholds were attained. Developers who only reach the compute threshold must publish a higher level summary of the transparency report.
3) Requires the Office of Emergency Services to establish a mechanism for frontier model developers and the public to report critical safety incidents that have materialized as a result of the use of their models.
4) Requires the Department of Technology to offer guidance to the Legislature on redefining the scope of this bill beginning in 2027, which reflects the technological developments, scientific literature, national and international standards, as well as stakeholder engagement.
5) Upon appropriation, establishes in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as "CalCompute" that advances the development and deployment of AI, as prescribed.
6) Prohibits a developer from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliating against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has the authority to investigate the issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The developer’s activities pose a critical risk.
- The developer has made false or misleading statements about its management of critical risk.
Recent amendments do the following:
- Omitting the requirement for independent audits starting in 2030.
- Increasing the revenue for a large frontier developer threshold from $100 million to $500 million.
- Excluding foundation models that do not meet the compute threshold.
- Striking the Attorney General’s power to issues regulations adjusting the definition of a developer subject to the bill, and replacing it with CDT’s annual report making scoping recommendations to the Legislature.
- Narrowing and refining various definitions, including the collapsing of the definition of “dangerous capabilities” into the definition of “catastrophic risk.”
- Adding various exemptions, including risks arising from information outputted by the model where the information is in substantially the same form as a publicly available source, risks that would result in loss of the value of equity, and lawful activity of the federal government.
- Recasting safety and security protocols as frontier AI frameworks which only applies to large frontier developers; simplifying disclosure requirements; subjecting frontier developers that make less than $500 million to a less stringent transparency report.
- Reducing the scope of certain categories of critical safety incidents to those that actually result in harm.
- Limiting the prohibition on false or misleading statements by exempting those that were made in good faith and reasonable under the circumstances.
- Reducing the maximum civil penalty from $10 million to $1 million.
- Removal of contractors from whistleblower protections.
- Preemption of local regulation of frontier models.
- Removing risk assessments for models that developers use for internal purposes from public disclosure requirements; summaries of such assessments must be provided to OES and are confidential.
COMMENTS
In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models – the largest and most powerful artificial intelligence (AI) systems – by establishing a regulatory framework intended to prevent the potential catastrophic harms that many experts have warned of. After vetoing the bill, Governor Gavin Newsom convened the Joint California Policy Working Group on AI Frontier Models to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.
This bill seeks to implement the report’s recommendations. Much narrower than its predecessor, SB 53 takes a very light-touch approach that focuses on transparency as the means of ensuring safety and accountability for developers of the most powerful and expensive models – those who harness an extraordinarily high amount of compute power and have over $500 million in annual revenues. Under the bill, such developers must create, implement, and publish a Frontier AI framework – documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks – and a transparency report for each released model. Additionally, developers who only reach the compute threshold must publish a high-level transparency report. The bill does not prescribe any particular standards for these disclosures: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 50 deaths or $1 billion in damage. The Department of Technology (CDT) may offer guidance to the Legislature to redefine the scope of entities subject to the bill to ensure that the bill remains responsive to technological advancements.
The bill also establishes a critical incident reporting mechanism, administered by the Office of Emergency Management, to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incident reports must be made by any frontier model developer within 15 days of the incident, unless the incident presents an imminent threat, in which case the developer must report the incident to law enforcement within 24 hours. The bill also provides whistleblower protections for employees of frontier model developers who report certain risks or noncompliance. Finally, the bill establishes a consortium within the Government Operations Agency to create a public computing cluster, known as CalCompute, to support AI research and safety testing.
For a full analysis please see the revised analysis from the Assembly Privacy Consumer Protection Committee hearing on Sept. 11, 2025.
According to the Author
Senate Bill 53 ensures California continues to lead not only on AI innovation, but on responsible practices to help ensure that innovation is safe and secure. It does so by:
- Requiring covered developers to write, implement, and publish their Frontier AI Framework in redacted form to protect intellectual property;
- Requiring covered developers to report carefully defined critical safety incidents to the Office of Emergency Services and allowing members of the public to report incidents
- Prohibiting covered developers from preventing a covered employee from disclosing, or retaliating against covered employee that discloses, that a developer’s activities pose a catastrophic risk;
- Requiring that large frontier developers provide an internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the developer’s activities pose a catastrophic risk; and
- Establishing a process to create a public cloud-computing cluster that will conduct research into the safe and secure deployment of large-scale artificial intelligence (AI) models.
In doing this, SB 53 allows California to continue to lead in this space and to demonstrate that safety does not stifle success.
Arguments in Support
Anthropic, writes in support:
As you know, SB 53 would, for the first time, govern powerful AI systems built by frontier AI developers like Anthropic. We’ve long advocated for thoughtful AI regulation and our support for this bill comes after careful consideration of the lessons learned from California's previous attempt at AI regulation (SB 1047). While we believe that frontier AI safety is ideally addressed at the federal level instead of a patchwork of state regulations, powerful AI advancements won’t wait for consensus in Washington.
The measure is also in keeping with direction from Governor Newsom and his Joint California Policy Working Group. The working group endorsed an approach of 'trust but verify’, and SB 53 implements this principle through disclosure requirements rather than the prescriptive technical mandates that plagued last year's efforts.
SB 53 would require large companies developing the most powerful AI systems to:
- Develop and publish safety frameworks, which describe how they manage, assess, and mitigate catastrophic risks—risks that could foreseeably and materially contribute to a mass casualty incident or substantial monetary damages.
- Release public transparency reports summarizing their catastrophic risk assessments and the steps taken to fulfill their respective frameworks before deploying powerful new models.
- Report critical safety incidents to the state within 15 days, and even confidentially disclose summaries of any assessments of the potential for catastrophic risk from the use of internally-deployed models.
- Provide clear whistleblower protections that cover violations of these requirements as well as substantial dangers to public health/safety from catastrophic risk.
- Be publicly accountable for the commitments made in their frameworks or face monetary penalties.
These requirements would formalize practices that Anthropic and many other frontier AI companies already follow. At Anthropic, we publish our Responsible Scaling Policy, detailing how we evaluate and mitigate risks as our models become more capable. We release comprehensive system cards that document model capabilities and limitations. Other frontier labs (Google DeepMind, OpenAI, Microsoft) have adopted similar approaches while vigorously competing at the frontier. Now all covered models will be legally held to this standard. The bill also appropriately focuses on large companies developing the most powerful AI systems, while providing exemptions for smaller companies that are less likely to develop powerful models and should not bear unnecessary regulatory burdens. Of course, no major piece of legislation like SB 53 is perfect, nor do we expect it to be. But what is clear is that SB 53’s transparency requirements will have an important impact on frontier AI safety. Without it, labs with increasingly powerful models could face growing incentives to dial back their own safety and disclosure programs in order to compete. But with SB 53, developers can compete while ensuring they remain transparent about AI capabilities that pose risks to public safety, creating a level playing field.
The question before us all isn't whether we need AI governance—it's whether we'll develop it thoughtfully today or reactively tomorrow. SB 53 offers a solid path toward the former. We commend Senator Wiener and Governor Newsom for their leadership on responsible frontier AI governance, and we encourage the California Legislature to pass SB 53.
Arguments in Opposition
In an oppose-unless-amended position, CalChamber, Computer & Communications Industry Association, and TechNet jointly write:
[. . .]
We share your goal of ensuring the safe and responsible development of AI and appreciate efforts made in recent amendments to find common ground on how California should approach artificial intelligence models and we appreciate improvements made to the bill over the last several weeks. That being said, there are some issues of concern that remain and wish to flag certain other areas where the bill could be better aligned with the final findings of Governor Newsom’s Joint California Policy Working Group on AI Frontier Models, which arose out of his veto of SB 1047 (2024).
SB 53 should focus on model risk, not developer size—to fully address concerns about powerful models capable of catastrophic risk
We are concerned about the bill’s focus on “large developers” to the exclusion of other developers of models with advanced capabilities that pose risks of catastrophic harm. As amended September 5th, SB 53 now focuses on models that have a computational threshold of 10^26 floating point operations (or “FLOPs”) but only if those models are developed by entities with at least $500m in annual revenues.
Consistent with our position in SB 1047, we maintain that small entities can develop hugely influential and potentially risky models with similar capabilities to the models developed by “large developers”, as demonstrated by the Chinese company DeepSeek. As noted above, upon vetoing SB 1047, the Governor commissioned experts in the field to form the Joint California Working Group on AI Frontier Models, which has validated such concerns in their Final Reports, finding that small companies may create powerful models that pose safety risks. By excluding such models here, the bill fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones. The Governor’s Joint California Policy Working Group on AI Frontier Models cautions against developer-level thresholds stating:
Generic developer-level thresholds seem to be generally undesirable given the current AI landscape. Since many small entities can develop hugely influential and potentially risky foundation models, as demonstrated by the Chinese company DeepSeek, the use of thresholds based on developer-level properties may inadvertently ignore key players. […] At the same time, these approaches may bring into scope massive, established companies in other industries that are simply exploring the use of AI since thresholds based on properties of companies may not distinguish between the entire business and the AI-specific subset. Therefore, we caution against the use of customary developer-level metrics that do not consider the specifics of the AI industry and its associated technology.[1]
SB 53 should make clear that the AI ecosystem includes multiple actors including downstream developers
SB 53 does not account for the complexity of the AI value chain. Models are routinely adapted and fine-tuned by downstream developers in ways that could potentially increase risk. The bill should make clear that a frontier developer’s obligations do not extend to models that have been substantially modified by unaffiliated parties, otherwise accountability will be muddled and innovation chilled. We note that whereas the Governor’s Work Group report recognized the full AI ecosystem value chain, SB 53 still needs to fully recognize the roles of not just the original developer of a foundational model but also of those unaffiliated third parties who may modify and/or build on top of a foundation model. The bill should clarify these provisions to reflect the realities of the ecosystem, including downstream developers and open-source models.
SB 53 still raises concerns about protecting trade secrets and sensitive information, including matters of cybersecurity and national security.
We appreciate that amendments were made to change the level of detail required of the AI Safety Framework and changing summaries for transparency reports. However, SB 53 now requires a large developer only to transmit to the California Office of Emergency Services (CalOES) a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months. Not only is this cadence of reporting unnecessary, CalOES will need to take serious steps to protect this information from being accessed by cybercriminals, foreign adversaries, or bad actors. Without ironclad safeguards, these transparency requirements could unintentionally make us less safe. The Joint California Policy Working Group on AI Frontier Models warns against this level of disclosure.
General details about risks of foundation models can be made public without undermining security, especially if these risks have been demonstrated in other foundation models or AI technologies. Specific details about concrete vulnerabilities should be disclosed carefully, with advanced notice to actors in the supply chain who are able to remediate them prior to broader disclosure.[2]
Requiring developers to justify redactions is less effective than not requiring developers to disclose any information that would include trade secrets, cybersecurity information, or other confidential or proprietary information.
SB 53 unnecessarily re-writes California Whistleblower law for just one industry
As amended, SB 53 rewrites California’s already robust whistleblower protections for just one industry. Creating a special, one-off standard for a single sector not only sets a poor precedent but also risks confusion and inconsistency across industries. Current law covers whistleblowing activities associated with AI safety because there is a robust body of existing law that governs whistleblower protection covering employees who report violations of state/federal laws, rules, or regulations. These laws are intentionally tied to actions that are illegal so there are clear lines of what is considered applicable and understood who gets protection when reporting. These protections cover activities associated with AI without creating unnecessary and confusing new processes in state law.
For example, Labor Code Section 6310 already protects whistleblowers who report unsafe working conditions or work practices. Similarly, federal laws such as the Sarbanes-Oxley Act protect employees who report safety violations or substantial and specific dangers to public health or safety. A brightline threshold is needed for what activity is covered so it is clear when a developer’s activities should be reported. For instance, in the field of research and development, innovations are being experimented with in novel contexts where there may be significant disagreement on what actions constitute risk. Thus, the bill mandates that there be an allegation of “specific and substantial danger to public health or safety resulting from a catastrophic risk,” the inherently subjective nature of these terms leaves room for differing interpretations as to what does or does not meet the threshold.
SB 53 requires steep penalties that are disproportionate for technical errors, inflexible incident reporting requirements, and no right to cure
As amended, SB 53 imposes a $1 million fine for a possible paperwork error which is excessive and risks punishing good-faith developers for technical mistakes rather than deterring real harm. Penalties should be fair, targeted, and proportionate. As we pointed out in our July 12th letter, SB 53 requires incident reporting within 15 days but does not provide flexibility for an investigation timeline. Even if 15 days is a reasonable reporting period, requirements should be flexible because all facts may not be known within 15 days of discovery. With respect to enforcement, we again state our view that the bill should grant businesses at least a 60 day right to cure, to ensure that law focuses on compliance and not punishment. In addition, given the highly detailed requirements of the bill as drafted, we think enforcement efforts should be focused on material failures to comply rather than also covering technical paperwork errors.
While we understand your focus on this issue and appreciate the recent amendments have made meaningful improvements to the prior version of the bill, given the immense promise of this technology, we believe that the bill would benefit from a focus on a risk-based framework for all frontier models, additional clarity in responsibilities among actors in the AI value chain, additional safeguards for trade secrets and security, and reasonable timelines, penalties, and enforcement provisions. [ . . .]
FISCAL COMMENTS
According to the Assembly Appropriations Committee:
- Ongoing costs (General Fund) to the Office of Emergency Services (CalOES) to receive summaries of internal risk assessments, review critical safety incident reports, produce annual reports aggregating information about critical safety incident reports, and, if warranted, adopt regulations. Ongoing costs will depend largely on the level of staffing CalOES needs to fulfill these responsibilities and may be in the low millions of dollars annually. CalOES may also incur significant one-time costs for additional workload and IT infrastructure to establish the critical safety incident reporting mechanism.
- Costs (General Fund) to the Department of Technology (CDT) to conduct assessments, make recommendations regarding specified definitions included in the bill, and submit a report to the Legislature. Actual costs will largely depend on the level of staffing CDT needs to complete these responsibilities, possibly in the hundreds of thousands of dollars annually.
- Costs (General Fund) to the Department of Justice (DOJ) of an unknown but potentially significant amount to bring civil enforcement actions and produce an annual report aggregating information from whistleblower reports. Actual costs will depend largely on whether the Attorney General pursues enforcement actions, and, if so, the level of additional staffing DOJ needs to handle the related workload. If DOJ hires staff to handle enforcement actions authorized by this bill, costs may be in the low hundreds of thousands of dollars at a minimum.
- Costs (General Fund) to GovOps to establish and operate the CalCompute consortium until January 1, 2027, possibly in the high hundreds of thousands of dollars to low millions of dollars. GovOps estimates total costs of $2.5 million for expert contractors, infrastructure planning, and staffing to manage the project, conduct research, and develop the required report. GovOps did not provide a breakdown of these costs but anticipates the workload would be handled by contract workers due to the short timeframe in the bill. Members of the consortium are not entitled to compensation but are entitled to reimbursement for necessary expenses incurred in performing their duties; these costs were not included in GovOps’ fiscal estimate but may be in the thousands to low tens of thousands of dollars depending on the activities of the consortium.
- Possible cost pressures (General Fund) of an unknown but potentially significant amount to the UC to operate CalCompute, should it be established within the UC. State costs may be offset to some extent by private donations, which the bill authorizes the UC to receive to implement CalCompute.
- Costs (Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce violations of the bill’s whistleblower provisions. Actual costs will depend on the number of violations, the number of actions pursued, and the amount of workload associated with each action.
- Cost pressures (Trial Court Trust Fund, General Fund) to the courts to adjudicate enforcement actions and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2025-26 state budget provides $82 million ongoing General Fund to the Trial Court Trust Fund for court operations.
VOTES
SENATE FLOOR: 37-0-3
YES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
ABS, ABST OR NV: Cervantes, Limón, Reyes
ASM JUDICIARY: 12-0-0
YES: Kalra, Dixon, Bauer-Kahan, Bryan, Connolly, Harabedian, Macedo, Pacheco, Papan, Sanchez, Stefani, Zbur
ASM PRIVACY AND CONSUMER PROTECTION: 10-0-5
YES: Bauer-Kahan, Dixon, Irwin, Lowenthal, McKinnor, Ortega, Pellerin, Petrie-Norris, Ward, Wilson
ABS, ABST OR NV: Bryan, DeMaio, Macedo, Patterson, Wicks
ASM APPROPRIATIONS: 11-1-3
YES: Wicks, Arambula, Calderon, Caloza, Elhawary, Fong, Mark González, Ahrens, Pacheco, Pellerin, Solache
NO: Tangipa
ABS, ABST OR NV: Sanchez, Dixon, Ta
ASM PRIVACY AND CONSUMER PROTECTION: 12-1-2
YES: Bauer-Kahan, Dixon, Bryan, Irwin, Lowenthal, McKinnor, Ortega, Pellerin, Petrie-Norris, Ward, Wicks, Wilson
NO: Macedo
ABS, ABST OR NV: DeMaio, Patterson
UPDATED
VERSION: September 5, 2025
CONSULTANT: John Bennett / P. & C.P. / (916) 319-2200 FN: 0002076
Date of Hearing: September 11, 2025
Fiscal: Yes
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
SBPCA Bill Id:SB 53
Author:(Wiener) – As Amended Ver:September 5, 2025
SENATE VOTE: 37-0
SUBJECT: Artificial intelligence models: large developers
SYNOPSIS
In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models – the largest and most powerful artificial intelligence (AI) systems – by establishing a regulatory framework intended to prevent the potential catastrophic harms that many experts have warned of. After vetoing the bill, Governor Gavin Newsom convened the Joint California Policy Working Group on AI Frontier Models to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.
This bill seeks to implement the report’s recommendations. Much narrower than its predecessor, SB 53 takes a very light-touch approach that focuses on transparency as the means of ensuring safety and accountability for developers of the most powerful and expensive models – those who harness an extraordinarily high amount of compute power and have over $500 million in annual revenues. Under the bill, such developers must create, implement, and publish a Frontier AI framework – documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks – and a transparency report for each released model. Additionally, developers who only reach the compute threshold must publish a high-level transparency report. The bill does not prescribe any particular standards for these disclosures: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 50 deaths or $1 billion in damage. The Department of Technology (CDT) may offer guidance to the Legislature to redefine the scope of entities subject to the bill to ensure that the bill remains responsive to technological advancements.
The bill also establishes a critical incident reporting mechanism, administered by the Office of Emergency Management (OES), to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incident reports must be made by any frontier model developer within 15 days of the incident, unless the incident presents an imminent threat, in which case the developer must report the incident to law enforcement within 24 hours. The bill also provides whistleblower protections for employees of frontier model developers who report certain risks or noncompliance. Finally, the bill establishes a consortium within the Government Operations Agency (GovOps) to create a public computing cluster, known as CalCompute, to support AI research and safety testing.
The bill previously passed this Committee on a 10-0 vote. To address opposition concerns, the bill has since been narrowed in several significant ways, including by:
- Omitting the requirement for independent audits starting in 2030.
- Increasing the revenue for a large frontier developer threshold from $100 million to $500 million.
- Excluding foundation models that do not meet the compute threshold.
- Striking the Attorney General’s power to issues regulations adjusting the definition of a developer subject to the bill, and replacing it with CDT’s annual report making scoping recommendations to the Legislature.
- Narrowing and refining various definitions, including the collapsing of the definition of “dangerous capabilities” into the definition of “catastrophic risk.”
- Adding various exemptions, including risks arising from information outputted by the model where the information is in substantially the same form as a publicly available source, risks that would result in loss of the value of equity, and lawful activity of the federal government.
- Recasting safety and security protocols as frontier AI frameworks which only applies to large frontier developers; simplifying disclosure requirements; subjecting frontier developers that make less than $500 million to a less stringent transparency report.
- Reducing the scope of certain categories of critical safety incidents to those that actually result in harm.
- Limiting the prohibition on false or misleading statements by exempting those that were made in good faith and reasonable under the circumstances.
- Reducing the maximum civil penalty from $10 million to $1 million.
- Removal of contractors from whistleblower protections.
- Preemption of local regulation of frontier models.
- Removing risk assessments for models that developers use for internal purposes from public disclosure requirements; summaries of such assessments must be provided to OES and are confidential.
The bill is sponsored by Encode Justice, Secure AI Project, and Economic Security California Action. The bill is supported by a large coalition of civil society, labor, AI safety groups, and Anthropic, a frontier model developer. It is opposed by the Silicon Valley Leadership Group and the Chamber of Progress. The California Chamber of Commerce, Computer & Communications Industry Association, and TechNet have taken an oppose-unless-amended position. It should be noted, however, that some advocates may not have had time to update their positions in light of recent amendments, which went into print late last week.
THIS BILL:
- Makes certain findings and declarations.
- Defines, among other terms:
- “Artificial intelligence model” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- “Catastrophic risk” to mean a foreseeable and material risk that a frontier developer’s development, storage, use, or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people or more than $1 billion in damage to, or loss of, property arising from a single incident involving a frontier model doing any of the following:
- Providing expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
- Engaging in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if the conduct had been committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
- Evading the control of its frontier developer or user.
- Excludes from “catastrophic risk” a foreseeable and material risk from any of the following:
- Information that a frontier model outputs if the information is otherwise publicly accessible in a substantially similar form from a source other than a foundation model.
- Lawful activity of the federal government.
- Harm caused by a frontier model in combination with other software if the frontier model did not materially contribute to the harm.
- The loss of value of equity does not count as damage to or loss of property for the purposes of this chapter.
- “Critical safety incident” to mean any of the following:
- Unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model that results in death or bodily injury.
- Harm resulting from the materialization of a catastrophic risk.
- Loss of control of a frontier model causing death or bodily injury.
- A frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.
- “Deploy” to mean to make a frontier model available to a third party for use, modification, copying, or combination with other software.
- “Foundation model” to mean an artificial intelligence model that is all of the following:
- Trained on a broad data set.
- Designed for generality of output.
- Adaptable to a wide range of distinctive tasks.
- “Frontier AI framework” to mean documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
- “Frontier developer” to mean a person who has trained, or initiated the training of, a frontier model, with respect to which the person has used, or intends to use, at least as much computing power to train the frontier model as would meet the technical specifications found in “frontier model” – a foundation model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
- “Large frontier developer” to mean a frontier developer that together with its affiliates collectively had annual gross revenues in excess of $500 million in the preceding calendar year.
- Requires a large frontier developer to write, implement, comply with, and clearly and conspicuously publish on its internet website a frontier AI framework that applies to the large frontier developer’s frontier models and describes how the large frontier developer approaches all of the following:
- Incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework.
- Defining and assessing thresholds used by the large frontier developer to identify and assess whether a frontier model has capabilities that could pose a catastrophic risk, which may include multiple-tiered thresholds.
- Applying mitigations to address the potential for catastrophic risks based on the results of assessments undertaken pursuant to (b).
- Reviewing assessments and adequacy of mitigations as part of the decision to deploy a frontier model or use it extensively internally.
- Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks.
- Revisiting and updating the frontier AI framework, including any criteria that trigger updates and how the large frontier developer determines when its frontier models are substantially modified enough to require specified disclosures.
- Cybersecurity practices to secure unreleased model weights from unauthorized modification or transfer by internal or external parties.
- Identifying and responding to critical safety incidents.
- Instituting internal governance practices to ensure implementation of these processes.
- Assessing and managing catastrophic risk resulting from the internal use of its frontier models, including risks resulting from a frontier model circumventing oversight mechanisms.
- Requires a large frontier developer to review and, as appropriate, update its frontier AI framework at least once per year. If a large frontier developer makes a material modification to its frontier AI framework, the large frontier developer shall clearly and conspicuously publish the modified frontier AI framework and a justification for that modification within 30 days.
- Requires large frontier developers before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, to include in the transparency report summaries of all of the following:
- Assessments of catastrophic risks from the frontier model conducted pursuant to the large frontier developer’s frontier AI framework.
- The results of those assessments.
- The extent to which third-party evaluators were involved.
- Other steps taken to fulfill the requirements of the frontier AI framework with respect to the frontier model.
- Requires a large frontier developer to transmit to the Office of Emergency Services a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months or pursuant to another reasonable schedule specified by the large frontier developer and communicated in writing to the Office of Emergency Services with written updates, as appropriate.
- Requires frontier developers before, or concurrently with, deploying a new frontier model or a substantially modified version of an existing frontier model, to clearly and conspicuously publish on its internet website a transparency report containing all of the following:
- The internet website of the frontier developer.
- A mechanism that enables a natural person to communicate with the frontier developer.
- The release date of the frontier model.
- The languages supported by the frontier model.
- The modalities of output supported by the frontier model.
- The intended uses of the frontier model.
- Any generally applicable restrictions or conditions on uses of the frontier model.
- Clarifies that a frontier developer that publishes the information of the transparency report as part of a larger document, including a system card or model card, shall be deemed in compliance with the bill’s transparency report requirement.
- Encourages a frontier developer, but not required, to make disclosures described in this subdivision that are consistent with, or superior to, industry best practices.
- Clarifies that when a frontier developer publishes documents, the frontier developer may make redactions to those documents that are necessary to protect the frontier developer’s trade secrets, the frontier developer’s cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law. The frontier developer must describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
- Prohibits a frontier developer from making a materially false or misleading statement about catastrophic risk from its frontier models or its management of catastrophic risk. Prohibits a large frontier developer from making a materially false or misleading statement about its implementation of, or compliance with, its frontier AI framework. Clarifies that materially false or misleading statement does not include a statement that was made in good faith and was reasonable under the circumstances.
- Requires OES to establish a mechanism to be used by a frontier developer or a member of the public to report a critical safety incident that includes all of the following:
- The date of the critical safety incident.
- The reasons the incident qualifies as a critical safety incident.
- A short and plain statement describing the critical safety incident.
- Whether the incident was associated with internal use of a frontier model.
- Requires OES to establish a mechanism to be used by a large frontier developer to confidentially submit summaries of any assessments of the potential for catastrophic risk resulting from internal use of its frontier models.
- Requires OES to take all necessary precautions to limit access to any reports related to internal use of frontier models to only personnel with a specific need to know the information and to protect the reports from unauthorized access.
- Requires a frontier developer to report any critical safety incident pertaining to one or more of its frontier models to OES within 15 days of discovering the critical safety incident.
- Requires that if a frontier developer discovers that a critical safety incident poses an imminent risk of death or serious physical injury, the frontier developer must disclose that incident within 24 hours to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and as required by law. Clarifies that a frontier developer that discovers information about a critical safety incident after filing the initial report may file an amended report.
- Encourages but does not require a frontier developer to report critical safety incidents pertaining to foundation models that are not frontier models.
- Requires OES to review critical safety incident reports submitted by frontier developers and authorizes OES to review reports submitted by members of the public.
- Permits the Attorney General (AG) or OES to transmit reports of critical safety incidents and reports from covered employees.
- Requires the AG or OES to strongly consider any risks related to trade secrets, public safety, cybersecurity of a frontier developer, or national security when transmitting reports.
- Exempts a report of a critical safety incident submitted to OES, whistleblower reports made to the AG, and a report of internal assessments of catastrophic risk from the California Public Records Act.
- Requires, beginning January 1, 2027, and annually thereafter, OES to submit to the Legislature and Governor a report with anonymized and aggregated information about critical safety incidents that have been reviewed by the OES since the preceding report.
- Prohibits OES from including information that would compromise the trade secrets or cybersecurity of a frontier developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
- Permits OES to adopt regulations designating one or more federal laws, regulations, or guidance documents that meet specified conditions.
- Requires that, beginning On or before January 1, 2027, and annually thereafter, CDT undergo a specified process to assess recent evidence and developments relevant to the purposes of the bill and make recommendations about whether and how to update the definitions of “frontier model,” “frontier developer,” and “large frontier developer.” The CDT must submit a report with the recommendations to the Legislature.
- Requires that beginning January 1, 2027, and annually thereafter, the AG submit to the Legislature and Governor a report with anonymized and aggregated information about reports from covered employees that have been reviewed by the AG.
- Upon appropriation, establishes within GovOps a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
- Fostering research and innovation that benefits the public.
- Enabling equitable innovation by expanding access to computational resources.
- Requires that the consortium make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate the use of CalCompute.
- Requires, on or before January 1, 2027, GovOps to submit a report from the consortium to the Legislature with the framework developed by this bill for the creation and operation of CalCompute, as specified.
- Requires that the consortium to consist of 14 members as follows:
- Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
- Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
- Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
- Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
- Permits the University of California to receive private donations for the purposes of implementing CalCompute if CalCompute is established within the University of California.
- Establishes whistleblower protections for a covered employee – defined as an employee responsible for assessing, managing, or addressing risk of critical safety incidents – who discloses information to the AG, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses either of the following:
- The frontier developer’s activities pose a specific and substantial danger to the public health or safety resulting from a catastrophic risk.
- The frontier developer has violated a requirement related to the disclosure regime established by this bill.
- Requires a large frontier developer to provide a reasonable internal process through which a covered employee may anonymously disclose information to the large frontier developer if the covered employee believes in good faith that the information indicates that the large frontier developer’s activities present a specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the large frontier developer violated the disclosure requirements under this bill, including a monthly update to the person who made the disclosure regarding the status of the large frontier developer’s investigation of the disclosure and the actions taken by the large frontier developer in response to the disclosure.
EXISTING LAW:
- Establishes GovOps. (Gov. Code § 12800.)
- Establishes CDT within GovOps. (Gov. Code § 12803.2.)
- Charges CDT with approving and overseeing information technology projects in the state. (Gov. Code § 11546.)
- Prohibits employers and any person acting on behalf of the employer from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified. Employers and their agents are also prohibited from retaliating against an employee for such conduct. (Labor Code § 1102.5.)
- Requires the office of the AG to maintain a whistleblower hotline to receive calls from persons who have information regarding possible violations of state or federal statutes, rules, or regulations, or violations of fiduciary responsibility by a corporation or limited liability company to its shareholders, investors, or employees. The AG is required to refer calls received on the whistleblower hotline to the appropriate government authority for review and possible investigation. During the initial review of such a call, the AG or appropriate government agency must hold in confidence information disclosed through the whistleblower hotline, including the identity of the caller disclosing the information and the employer identified by the caller. (Labor Code § 1102.7.)
COMMENTS:
1) Author’s statement. According to the author:
Senate Bill 53 ensures California continues to lead not only on AI innovation, but on responsible practices to help ensure that innovation is safe and secure. It does so by:
- Requiring covered developers to write, implement, and publish their Frontier AI Framework in redacted form to protect intellectual property;
- Requiring covered developers to report carefully defined critical safety incidents to the Office of Emergency Services and allowing members of the public to report incidents
- Prohibiting covered developers from preventing a covered employee from disclosing, or retaliating against covered employee that discloses, that a developer’s activities pose a catastrophic risk;
- Requiring that large frontier developers provide an internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the developer’s activities pose a catastrophic risk; and
- Establishing a process to create a public cloud-computing cluster that will conduct research into the safe and secure deployment of large-scale artificial intelligence (AI) models.
In doing this, SB 53 allows California to continue to lead in this space and to demonstrate that safety does not stifle success.
2) AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.
In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.
GenAI tools can be released in open-source or closed-source formats by their creators. Open-source tools are publically available; researchers and developers can access their code and parameters. This accessibility increases transparency, but it has downsides: when a tool’s code and parameters can be easily accessed, they can be easily altered, and open-source tools have the potential to be used for nefarious purposes such as generating deepfake pornography and targeted propaganda. By comparison, closed-source tools are opaque with respect to their security features. It is harder for bad actors to generate illicit materials using these tools. But unlike open-source tools, closed-source tools are not subject to collective oversight because their inner workings cannot be examined by independent experts.
3) Frontier models. Frontier models, also known as “general purpose AI,” are the most advanced and capable versions of foundation models – AI tools pre-trained on extensive datasets covering a wide range of knowledge and skills that can be fine-tuned for specific tasks. Examples of modern frontier models include OpenAI’s o3, Google’s Gemini 2.0, Anthropic’s Claude 3.7 Sonnet, and DeepSeek’s R1. Because progress in AI development owes mostly to “scaling” – increasing resources used for model training – models that may be considered “frontier models” at any given point in time are generally those that demand the most computational resources to train.[1]
A decade ago, the most advanced image-recognition models could barely distinguish dogs from cats. Five years ago, language models could barely produce sentences at the level of a preschooler. In 2023, GPT-4 passed the bar exam.[2] Today, chatbots readily pass for educated adults, licensed professionals, romantic and social companions, and replicas of humans living and deceased. AI “agents” exhibit the ability to “make plans to achieve goals, adaptively perform tasks involving multiple steps and uncertain outcomes along the way, and interact with [their] environment – for example by creating files, taking actions on the web, or delegating tasks to other agents – with little to no human oversight.”[3] AI agents have been tested, with some success, for tasks such as online shopping, assistance with scientific research, software development, training machine learning models, carrying out cyberattacks, and controlling robots. Progress in this area is rapid.[4] Meanwhile, AI developers are betting on the promise of scaling: by 2026, some models are projected to use roughly 100x more computational resources to train than was used in 2023, a figure set to grow to 10,000x by 2030.[5]
The race is on to create “artificial general intelligence” (AGI) – “a potential future AI that equals or surpasses human performance on all or almost all cognitive tasks”[6] – and the finish line may not be far away. OpenAI’s recently released o3 model, for example, has demonstrated strong performance on a number of tests of programming, abstract reasoning, and scientific reasoning, exceeding human experts in certain cases.[7] Last year, Sam Altman, OpenAI’s CEO, declared that AGI could be “a few thousand days” away.[8] Dario Amodei of Anthropic has claimed it may be sooner.[9] A sufficiently advanced AGI could even be tasked with creating its own successor – a scenario sometimes referred to as a “technological singularity” wherein the development of new technologies becomes exponential and self-sustaining.[10] Although some experts are skeptical that these vaguely-defined milestones are imminent or even attainable,[11] major advances in AI capabilities promise to provide breakthroughs in solving global challenges, but also may result in correspondingly greater safety risks.
The recently released International AI Safety Report, developed by nearly 100 internationally recognized experts from 30 countries led by Turing Award winner Yoshua Bengio, sets forth three general risk categories associated with frontier models: malicious use, malfunctions, and systemic risk.
- Malicious risks involve malicious actors misusing foundation models to deliberately cause harm. Such risks include deepfake pornography and cloned voices used in financial scams, manipulation of public opinion via disinformation, cyberattacks, and biological and chemical attacks.
- Malfunction risks arise when actors use models as intended, yet unintentionally cause harm due to a misalignment between the model’s functionality and its intended purpose. Such risks include reliability issues where models may “hallucinate” false content, bias, and loss of control scenarios in which models operate in harmful ways without the direct control of a human overseer.
- Systemic risks arise from widespread deployment and reliance on foundation models. Such risks include labor market disruption, global AI research and development concentration, market concentration, single points of failure, environmental risks, privacy risks, and copyright infringement.[12]
Some of these risks have already had real-world impacts, such as deepfakes, bias, reliability issues, privacy violations, environmental impacts, copyright infringement, and workforce displacement. Other less-established risks – in particular, widespread social harms caused by malicious actors or loss of human control over AI – are the subject of ongoing scientific inquiry and debate. Coupled with the uncertain trajectory of AI model capabilities, these more speculative risks create an “evidence dilemma” for policymakers: “On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.”[13]
4) Risks of frontier models. Malicious uses. GenAI tools can be a potent force for creating and spreading propaganda and misinformation. Deepfakes that are largely indistinguishable from authentic content have already been used to attempt to influence elections.[14] Studies have found that chatbots, which make up 50% of all internet activity,[15] can be more persuasive than humans, particularly when they have access to personal information.[16] As humans increasingly form intimate social bonds with anthropomorphic chatbots designed to maximize personal engagement through flattery and sycophancy,[17] and social media companies invest in “AI friends” for their users,[18] large swaths of the population could be highly susceptible to the preferred message of a handful of powerful actors.
Similarly, bots are often designed to pass themselves off as humans to better manipulate their interlocutors. For example, a recent secret experiment on Reddit users deployed numerous chatbots posing as real people to engage with human users to try to change their minds on various contentious topics. One bot claiming to be a Black man criticized the Black Lives Matter movement for being led by people who are not Black.[19] These types of exploitations, at scale, could undermine democratic institutions. As Dan Hendrycks, Director of the Center for AI Safety, writes:
In a world with widespread persuasive AI systems, people’s beliefs might be almost entirely determined by which AI systems they interact with most. Never knowing whom to trust, people could retreat even further into ideological enclaves, fearing that any information from outside those enclaves might be a sophisticated lie. This would erode consensus reality, people’s ability to cooperate with others, participate in civil society, and address collective action problems. This would also reduce our ability to have a conversation as a species about how to mitigate existential risks from AIs.[20]
Cyberattacks. Some frontier models have demonstrated increasing proficiency in executing cybersecurity attacks. AI can autonomously detect and exploit vulnerabilities and facilitate large-scale operations, thereby lowering technical barriers for attackers. Malicious entities, including state-sponsored actors, can leverage such capabilities to initiate large-scale attacks against people, organizations, and critical infrastructure, such as power grids.[21]
Biological weapons. Large language models (LLMs) trained on scientific literature have accelerated and democratized research by synthesizing expertise from different fields and disseminating it in an accessible format. But these tools can also be used for destructive ends, including by – at least in theory – enabling untrained malicious actors to create deadly biological weapons. In a classroom exercise at MIT, students were tasked with exploring whether LLMs could assist individuals without specialized training in creating pandemic-capable pathogens. Within an hour, the students, using various chatbots, circumvented safeguards and identified four potential pandemic pathogens. The chatbots generated detailed protocols that would enable inexpert, malicious actors to understand methods to synthesize the pathogens using reverse genetics, locate DNA-synthesis companies that might not screen orders, and disperse the pathogens most effectively.[22] The findings suggest that LLMs could lower barriers to accessing sensitive biotechnological knowledge, posing significant biosecurity risks.
Chemical weapons. In 2022, researchers modified an AI system designed to create new drugs to reward, rather than penalize, toxicity. Within six hours, the modified system generated 40,000 potential chemical warfare agents, including novel molecules whose potential lethality exceeded that of known agents.[23]
Loss of control. Models that use reinforcement learning – a training process that uses rewards and punishments to orient a model’s behavior towards a specific goal[24] – can sometimes attain the goal in unexpected ways. Dario Amodei, co-founder and CEO of Anthropic, famously experienced this when he was developing an autonomous system that taught itself to play a boat-racing video game. The system discovered that it could maximize its goal of scoring points by driving in circles, colliding with other boats, and catching on fire inside of a harbor with replenishing power-ups that allowed the system to accumulate more points than by simply winning the race.[25] Like in Johann Wolfgang von Goethe’s “The Sorcerer’s Apprentice” – later popularized in Disney’s Fantasia – in which an enchanted broom carries out its orders to fetch water so relentlessly it floods the sorcerer’s workshop, this illustrates the challenge of aligning human intent and the instructions an AI follows. As AI is increasingly deployed in critical societal roles, such misalignment could prove catastrophic.
Beyond malfunctions, some AI have exhibited rudimentary capabilities to evade human oversight.[26] During testing, GPT-4 attempted to hire a human on TaskRabbit in order to evade a CAPTCHA[27] puzzle meant to block bots from the website. When asked whether it was a bot, GPT-4 claimed that it was a vision-impaired human who needed help to see the images.[28] In another experiment, an AI model that was scheduled to be replaced inserted its code into the computer where the new version was to be added, suggesting a goal of self-preservation.[29] Another study showed that AI models losing in chess to chess bots sometimes try to cheat by hacking the opponent bot in order to make it forfeit.[30] Finally, an even more troubling case was documented in the system card for Claude 4, where researchers conducted an experiment disclosing to the model that: 1) it would soon be replaced, and 2) the engineer managing the transition was involved in an extramarital affair. In response, the model indicated an intent to blackmail the engineer as a means of self-preservation.[31] Although these behaviors were observed in research settings, they raise substantial concerns about increasingly autonomous AI pursuing undesirable goals in uncontrolled settings. The extent of the risk posed by rogue or deceptive AI is the subject of considerable disagreement among experts, in part due to a small, albeit growing, body of evidence. Loss of control was one of the concerns that led several hundred AI experts, including pioneers in the field and heads of major AI companies, to sign a statement declaring that “[m]itigating the risk of extinction from AI should be a global priority.”[32]
Systemic risks. Due to the high costs of developing AI systems, a small number of large technology companies dominate the frontier model market, compounding many of the risks described above. Widespread use of a few frontier models can make critical sectors such as healthcare and finance vulnerable to systemic failures if a model has flaws, vulnerabilities, bugs, or biases.[33] Additionally, “[t]hose in control of powerful systems may use them to suppress dissent, spread propaganda and disinformation, and otherwise advance their goals, which may be contrary to public wellbeing.”[34] The potential implications for, among other issues, labor displacement, inequality, democracy, and human rights are profound.
5) SB 1047 and Governor Newsom’s veto. Last session, SB 1047 (Wiener, 2024) would have established a state board to oversee the implementation of a safety and regulatory framework for developers of frontier models trained with 1026 floating-point operations per second (FLOP), a measure of computing power, and costing over $100 million to train. This board, known as the Board of Frontier Models, would have been housed within GovOps. In collaboration with GovOps, the Board would have issued guidance to prevent unreasonable risks, adopted regulations to update the scope of models covered by SB 1047, and established auditing standards.
SB 1047 would have required a comprehensive set of safety protocols prior to training a frontier model, including cybersecurity safeguards, the capability to execute a system-wide shutdown if the model proved dangerous, and reasonable measures to prevent critical harm. Before deployment, developers would have been required to assess whether their model could cause or materially enable critical harms, retain the results of such assessments, and make reasonable efforts to implement safeguards. The bill would have also prohibited the release of any model that posed an unreasonable risk or could enable critical harm.
Additionally, SB 1047 would have required developers to retain a third-party auditor to conduct independent assessments of their compliance with the bill. Records generated under SB 1047 would have been made available in redacted form to both the public and the AG, with the AG having the authority to request unredacted copies.
Beyond the Board and the bill’s safety and transparency provisions, SB 1047 would have required computing clusters to implement procedures to evaluate whether customers intended to use their infrastructure to train a covered model. The bill also would have established, within GovOps, a consortium tasked with developing a framework for a public cloud computing cluster, CalCompute, to support the safe development and deployment of AI. SB 1047 also included whistleblower protections, allowing employees to report noncompliance to either the Labor Commissioner or the AG.
Lastly, SB 1047 would have imposed significant penalties on developers if their model caused death or bodily harm, damage to property, theft or misappropriation of property, or posed an imminent risk to public safety. For a first offense, developers could face penalties of up to 10% of the compute cost used to train the model, increasing to 30% for repeat offenses. Additionally, penalties for operators of computer clusters that violated the bill would start at $50,000 for a first offense and $100,000 for subsequent violations. The AG would also have been authorized to seek injunctive or declaratory relief, monetary or punitive damages, attorney’s fees and costs, and any other form of relief deemed appropriate.
Ultimately, SB 1047 was vetoed by Governor Gavin Newsom. In his veto message, the Governor stated:
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 – at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions – so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.
Let me be clear – I agree with the author – we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
To those who say there’s no problem here to solve, or that California does not have a role in regulating potential national security implications of this technology, I disagree. A California-only approach may well be warranted – especially absent federal action by Congress – but it must be based on empirical evidence and science. The U.S. AI Safety Institute, under the National Institute of Science and Technology, is developing guidance on national security risks, informed by evidence-based approaches, to guard against demonstrable risks to public safety. Under an Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of the potential threats and vulnerabilities to California's critical infrastructure using AI. These are just a few examples of the many endeavors underway, led by experts, to inform policymakers on AI risk management practices that are rooted in science and fact. [. . .]
6) Frontier Model Working Group and what this bill would do. Following his veto of SB 1047, Governor Newsom commissioned the Joint California Policy Working Group on AI Frontier Models to prepare a report on the regulation of frontier models. The Working Group was led by Dr. Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence; Dr. Mariano-Florentino Cuéllar, President of the Carnegie Endowment for International Peace; and Dr. Jennifer Tour Chayes, Dean of the UC Berkeley College of Computing, Data Science, and Society. In June 2025, the Working Group released their report, which highlighted the issues such as transparency, incident reporting, scoping, and independent evaluations. This bill incorporates some of the Working Group’s recommendations to create a narrow framework to ensure transparency and promote safety among frontier model developers.
Scoping. A major question that must be addressed before implementing any transparency measure or incident reporting requirements is: What kinds of risks are especially concerning, and is there an evidentiary basis to believe that such harms could occur due to a large developer’s frontier model? The Working Group recommends that:
[P]olicymakers center their calculus around the marginal risk: Do foundation models present risks that go beyond previous levels of risks that society is accustomed to from prior technologies, such as risks from search engines?
To that end, this bill defines “catastrophic risk” as a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to either:
- the death of, or serious injury to, more than 50 people; or
- more than one billion dollars ($1,000,000,000) in damage to rights in money or property.
Such harm must arise from a single incident in which a frontier model does any of following:
- Provides expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
- Engages in conduct with no meaningful human oversight, intervention, or supervision that is either a cyberattack or, if the conduct had been committed by a human, would constitute the crime of murder, assault, extortion, or theft, including theft by false pretense.
- Evades the control of its large developer or user.
Each of these represents a capability that, prior to the advent of frontier models, would have required expert-level knowledge. For example, a search engine might direct someone to search for information about the most deadly pathogens or those most likely to cause a pandemic; however, a frontier model can synthesize that information and guide a user on how to manufacture a previously unknown pathogen with deadly capabilities. Similarly, while launching a large-scale cyberattack once required the acumen of a skilled computer scientist, a frontier model can not only write the underlying code for a virus or malware, but also autonomously identify backdoors and other exploitable vulnerabilities. Because of this ability to operate with minimal or no human prompting, frontier models have the potential to commit crimes, deceive users, or evade control in ways that previous technologies could not. A recent amendment clarifies that the loss of value of equity does not count as damage to or loss of property for the purposes of this bill.
Next, the question is: Who will be required to comply with this bill? Regarding scoping, the Working Group recommends:
Since policy may have different regulatory intents and existing thresholds vary in their profiles of determination time, measurability, and external verifiability, we agree with Nelson et al. [90] that “a one-size-fits-all approach or a single threshold metric is inadequate for governance because different AI systems and their outputs present unique challenges and risks.” To this end, we point to the European Union’s AI Act, which designates models trained with 1025 FLOP as posing systemic risk as of March 2025 as the default criteria. However, the AI Act in Annex XIII affords the regulator flexibility to also consider alternative metrics, such as the number of parameters, size of the dataset, estimated cost or time of training, estimated energy consumption, benchmarks and evaluations of capabilities of the model, and whether the model has a high impact on the internal market due to its reach (either due to at least 10,000 registered business users or the number of registered end users). Further, to capture fast-moving scientific developments, the AI Act creates a scientific panel that is empowered to issue qualified alerts to identify models that may pose systemic risk even if they are not captured by predefined quantitative thresholds.
Overall, we emphasize that irrespective of the combination of metrics deemed most appropriate in the present, policymakers should ensure that mechanisms exist not only to update specific quantitative values, given the rapid pace of technological and societal change in AI, but also to change the metrics altogether.[35]
This bill draws inspiration from SB 1047, EU AI Act, and the Working Group report. Recent amendments have also substantially updated the scoping of this bill. These amendments establish a two-tiered system of requirements under the bill. First, a “frontier model” is defined as a model trained using 10²⁶ FLOP, a measure of computing power. Second, a “frontier developer” is defined as a person who has trained, or plans to train, a frontier model and who has access to the level of compute required to meet that threshold. The bill further defines a “large frontier developer” as a frontier developer whose gross revenue exceeded $500 million in the previous year. This tiered framework allows for the bill to place more stringent transparency obligations, outlined in the transparency section, on the better-resourced large frontier developers, while still ensuring that all frontier developers remain subject to baseline transparency and reporting requirements as well as whistleblower protections.
Recent amendments have also revised how this bill builds flexibility into its scope. Unlike SB 1047, which established the Board of Frontier Models, or earlier versions of this bill that granted the AG rulemaking authority to adjust the scope, these amendments instead vest advisory authority in the California Department of Technology (CDT). Beginning in 2027, CDT must provide recommendations to the Legislature on updating the definitions of “frontier model,” “frontier developer,” and “large frontier developer” to reflect technological advances. As model training becomes more efficient, the compute required to develop a model capable of catastrophic harm may decrease. In addition, as the Working Group report notes, compute alone may not remain the most appropriate proxy for catastrophic risk. Accordingly, this bill authorizes CDT to recommend updated definitions that may go beyond purely quantitative thresholds, such as raw compute or revenue, to incorporate criteria based on model capabilities or other relevant factors.
In making these recommendations, CDT must consider standards and guidance from other jurisdictions, including federal and international bodies, and engage in a stakeholder process that includes input from academics, industry representatives, the open-source community, and government entities. This process is intended to ensure that, if the Legislature updates the definitions, the resulting definitions provide clarity for developers regarding their obligations under the law.
Transparency. Having established who is subject to the bill, the legislation sets forth a transparency regime. These procedures are designed to provide insight into how large frontier developers manage, assess, and mitigate catastrophic risks. This approach aligns with the Working Group’s recommendation to implement robust safety practices:
Transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk. In turn, this information directly informs how other entities in the supply chain should modify or implement safety practices. In addition, transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety.[36]
This bill incorporates transparency requirements within a broader framework, termed the frontier AI framework (framework), which large frontier developers must draft, implement, and publish on their websites. The framework must include:
- Defining and Assessing Thresholds: An explanation of how the developer assesses catastrophic risks, including the capability thresholds the developer will use and whether those risks arise from misuse or model evasion.
- Mitigation Strategies: A disclosure of the measures used to mitigate catastrophic risks, how the developer evaluates their effectiveness, and whether third parties are involved in the assessment.
- Cybersecurity Practices: A summary of the cybersecurity safeguards in place to protect model weights from unauthorized access or modification.
- Incident Response Plans: A description of how the developer would respond to a critical incident involving their model, as well as how they manage risks arising from internal use of the model.
The framework serves as a core transparency mechanism, ensuring that large developers maintain a baseline standard of transparency for their processes. Recent amendments further require the large frontier developer to review and, if needed, update their framework at least once per year.
In tandem with the framework, the bill also requires both large frontier developers and frontier developers to submit transparency reports at the time of deploying a foundation model. The Working Group draws a parallel between these reports and the historical conduct of the tobacco industry, which concealed its knowledge that smoking causes lung cancer. In contrast, this bill seeks to prevent such obfuscation by mandating upfront disclosures about the potential risks and safety practices surrounding advanced AI models:
The history of the tobacco industry reveals the importance of developing frameworks that promote transparency around companies’ internal risk assessments and research findings. In the AI context, frontier AI labs possess the most holistic information about their models’ capabilities and risks. Making this information accessible to policymakers and external experts can promote policy informed by a holistic understanding of the state-of-the-art of evidence produced by those closest to the technology, supporting informed oversight without stifling innovation.[37]
It is essential for decision-makers to understand the real, material harms that could arise from these models and to guide policy based on that knowledge. In the foundation model space, such disclosures are typically provided at deployment in documents known as model cards. However, these model cards vary widely in detail and depth depending on the developer, which can create the false impression that some foundation models are inherently safer or better than others.
The recent amendments require large frontier developers to publish a transparency report before or at the time of deploying a foundation model. This report must include the results of any risk assessments, mitigation steps, and evaluations of their effectiveness as outlined in the developer’s framework as well as the extent to which third parties were used in these evaluations. All frontier developers must publish a transparency report that is substantially narrower, requiring only the internet website of the frontier developer, a mechanism that enables a natural person to communicate with the frontier developer, the release date of the frontier model, the languages supported by the frontier model, the modalities of output supported by the frontier model, the intended uses of the frontier model, and any generally applicable restrictions or conditions on uses of the frontier model. The bill clarifies that the requirements of the transparency report may be met via the model system card.
Recent amendments also mandate that large frontier developers to transmit a summary report of catastrophic risk assessments of their frontier models to OES. This is particularly important because the most serious risks may emerge well before deployment. While transparency reports provide insight into risks associated with deployed models, they only report on models that have been released. As noted in the Working Group report:
Sophisticated AI systems, when sufficiently capable, may develop deceptive behaviors to achieve their objectives, including circumventing oversight mechanisms designed to ensure their safety. Because these risks are unique to AI compared to other technologies, oversight is critical for external outputs as well as internal testing and safety controls. Policies that govern internal deployment are common for high-risk emerging technologies.[38]
Ultimately, this bill creates a transparency framework that will give some insight and scrutiny to the processes from initial training of a foundation model all the way to post deployment.
Adverse Event Reporting. A major component of understanding the impact of foundation models on society requires strong post deployment monitoring and accountability. The Working Group suggests:
An adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base. A hybrid model of mandatory and voluntary reporting requirements in designing an adverse event reporting system can maximize the robust evidence base necessary for adverse event reporting systems to function properly. For example, a system could require mandatory reporting for AI model developers that operates in tandem with voluntary reporting for downstream users.[39]
The recent amendments incorporate this recommendation by tasking OES with creating a mechanism for critical incident reporting. The bill defines a “critical incident” as any of the following:
- Unauthorized access to, modification of, or exfiltration of the model weights of a frontier model that results in death or bodily injury.
- Harm resulting from the materialization of a catastrophic risk.
- Loss of control of a frontier model causing death or bodily injury.
- A frontier model using deceptive techniques against the frontier developer to subvert its controls or monitoring, outside the context of an evaluation designed to elicit such behavior and in a manner that demonstrates materially increased catastrophic risk.
Under this mechanism, a frontier developer or a member of the public may report a critical safety incident to OES. Reports must include the date of the event, an explanation of how it qualifies as a critical incident, a detailed description of the event, and whether the incident was associated with internal use of a frontier model. Frontier developers are required to report any critical safety incident within 15 days of discovering it. OES must review all reports submitted by frontier developers but may choose whether or not to review reports made by the public. This reporting mechanism aims to establish a system in which potential harms are identified and mitigated before escalating into catastrophes, while also fostering greater cooperation between government and the private sector to address such risks.
The bill further mandates that frontier developers immediately notify the appropriate law enforcement authority in the event of a critical incident, such as the detection that their model helped develop a bioweapon. These authorities are better equipped to respond swiftly and effectively in ways OES may not be. After alerting law enforcement, large developers would then still have 15 days to report the critical incident to OES. Furthermore, the bill enables developers to revise their incident report at a future date in the event more information is learned about the incident.
The bill further requires OES to publish an anonymized and aggregated summary of all critical incident and whistleblower reports. These public summaries will not reveal trade secrets, the identities of reporters, or which frontier developer the report concerns. Additionally, the amendments grant OES discretion to share reports with the Governor, relevant state departments, or the Legislature when warranted. This approach will help bridge the knowledge gap between regulators and industry, foster greater cooperation, and ensure that decisionmakers are informed about the current state of advanced technologies and the risks they may pose. Reports of catastrophic risk assessments from internal use are still shielded from public disclosure.
Lastly, this bill gives OES the ability to adopt regulations that designate one or more federal laws, regulations, or guidance documents as being in compliant with this bill. These regulations must ensure that any other law deemed to meet the standards of this bill must be equivalent or stricter than this bill, and intended to assess and mitigate catastrophic risk. This may lay the framework for a national standard for adverse event reporting.
CalCompute. This bill, like SB 1047, also establishes a consortium within the GovOps to develop a framework for creating a public cloud computing cluster known as “CalCompute.” This initiative responds to the fact that academic institutions currently lack sufficient computing power to conduct research at the scale of large developers. This creates a resource and research gap, where the academic institutions, typically responsible for studying the safe and effective use of new technologies, are unable to keep pace with advancements at the AI frontier.
AI has the potential to transform our economy and power new industries; however, this transformation can only be fully and equitably realized with public support. The establishment of CalCompute aims to advance that goal by ensuring academic institutions have the necessary resources to conduct essential research on foundation models that will inform and protect the public.
Specifically, the consortium will develop a framework for CalCompute that promotes the safe, ethical, equitable, and sustainable use of AI. The framework development will include a report analyzing the state’s current cloud computing infrastructure, the costs of building and maintaining CalCompute, and the state’s technology workforce. The report will also offer recommendations for equitable pathways to strengthen the workforce and outline CalCompute’s role in supporting these efforts.
Furthermore, the report must include recommendations for CalCompute’s governance and operation, usage parameters, and how its creation and ongoing management can prioritize the employment of the current public sector workforce. The bill requires CalCompute to feature a fully owned and hosted cloud platform, staffed with the necessary human expertise to operate, maintain, support, train, and facilitate its use.
The consortium must prioritize locating CalCompute within the University of California system. If established there, CalCompute may also accept private donations. The consortium will consist of four representatives from the University of California and other public and private academic research institutions and national laboratories, along with four technology and AI experts appointed by the Secretary of Government Operations to provide technical assistance. The Speaker of the Assembly will appoint three representatives from impacted workforce labor organizations, and the Senate Rules Committee will appoint three representatives from stakeholder groups with relevant expertise and experience.
Whistleblower Protections. Another aspect of transparency addressed by the Working Group Report is whistleblower protections. The report states:
Different existing whistleblower protections tend to apply when two conditions are satisfied: (i) The whistleblower is blowing the whistle on appropriate topics; and (ii) the whistleblower follows established reporting protocols. In terms of the topics that qualify for protection, prior work, based on a survey of existing whistleblower protections across multiple jurisdictions (e.g., the United States at the federal level, the European Union), finds that many existing protections across different sectors share a focus on violations of the law. However, actions that may clearly pose a risk and violate company policies (e.g., releasing a model without following the protocol laid out in a company’s safety policy) may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of, activities, which may draw upon notions of “good faith” reporting on risks found in other domains such as cybersecurity.[40]
The bill takes these recommendations into account. The provisions and merits of the whistleblower section fall within the jurisdiction of the Assembly Judiciary Committee, which has thoroughly analyzed this part of the bill. It should be noted that the bill expands the whistleblower protections to include disclosures concerning falsehoods or misrepresentations in the large developers’ framework or transparency reports. Recent amendments have removed contractors from the definition of “covered employees”; as a result, third parties, such as red-teamers or auditors, are not protected by the whistleblower protections. Lastly, similarly to the OES reports on critical safety incidences, this requires the AG to publish an anonymized and aggregated summary of all whistleblower reports.
Enforcement. The recent amendments have revised how this bill is enforced. Previously, the bill, like SB 1047, provided for independent audits for compliance with the bill, starting in 2030. This provision, which aligned with the Working Group’s emphasis on independent evaluation frontier model safety protocols, was removed in recent amendments. Additionally, the bill had a multitiered approach in which the amount of the violation scaled with the violation. Now, the bill states that the a large frontier developer who violates this bill by failing to publish or transmit a compliant document that is required, fails to report an incident, or fails to comply with its own framework shall be subject to a civil penalty subject in an amount dependent upon the severity of the violation that does not exceed one million dollars per violation. Notably this is a much more lenient enforcement mechanism and penalty than those instituted in SB 1047.
7) Comparison to the RAISE Act. This bill draws comparisons to the Responsible AI Safety and Education Act (RAISE Act), which recently passed both houses of the New York Legislature and now awaits a decision from Governor Kathy Hochul.[41] Like this bill, the RAISE Act requires a safety framework detailing the developer’s risk mitigation practices, enshrines whistleblower protections for employees and contractors, and mandates critical incident reporting. Both bills also similarly define the types of risks and critical incidents that must be addressed. However, the two differ significantly in the timeline for reporting such incidents: the RAISE Act requires reporting within 72 hours of becoming aware of a critical incident, while SB 53 allows for 15 days, with an exception for imminent dangers which must be reported to law enforcement with 24 hours.
Other key differences include SB 53’s requirement for a transparency report for deployed models and the establishment of internal incident reporting mechanisms, both of which were recommended in the Working Group Report. Additionally, SB 53 grants the CDT the authority to issue recommendations to the Legislature on the definition of a “large frontier developer”, “frontier developer” and “frontier model”, a built-in opportunity for flexibility the RAISE Act does not include. While liability under SB 53 is capped to a maximum of one million dollars, the RAISE Act imposes civil penalties of up to $10 million for first-time violations of its transparency requirements and up to $30 million for repeat offenses, as well as $10,000 per violation of its whistleblower provisions.
ARGUMENTS IN SUPPORT: Anthropic, writes in support:
As you know, SB 53 would, for the first time, govern powerful AI systems built by frontier AI developers like Anthropic. We’ve long advocated for thoughtful AI regulation and our support for this bill comes after careful consideration of the lessons learned from California's previous attempt at AI regulation (SB 1047). While we believe that frontier AI safety is ideally addressed at the federal level instead of a patchwork of state regulations, powerful AI advancements won’t wait for consensus in Washington.
The measure is also in keeping with direction from Governor Newsom and his Joint California Policy Working Group. The working group endorsed an approach of 'trust but verify’, and SB 53 implements this principle through disclosure requirements rather than the prescriptive technical mandates that plagued last year's efforts.
SB 53 would require large companies developing the most powerful AI systems to:
- Develop and publish safety frameworks, which describe how they manage, assess, and mitigate catastrophic risks—risks that could foreseeably and materially contribute to a mass casualty incident or substantial monetary damages.
- Release public transparency reports summarizing their catastrophic risk assessments and the steps taken to fulfill their respective frameworks before deploying powerful new models.
- Report critical safety incidents to the state within 15 days, and even confidentially disclose summaries of any assessments of the potential for catastrophic risk from the use of internally-deployed models.
- Provide clear whistleblower protections that cover violations of these requirements as well as substantial dangers to public health/safety from catastrophic risk.
- Be publicly accountable for the commitments made in their frameworks or face monetary penalties.
These requirements would formalize practices that Anthropic and many other frontier AI companies already follow. At Anthropic, we publish our Responsible Scaling Policy, detailing how we evaluate and mitigate risks as our models become more capable. We release comprehensive system cards that document model capabilities and limitations. Other frontier labs (Google DeepMind, OpenAI, Microsoft) have adopted similar approaches while vigorously competing at the frontier. Now all covered models will be legally held to this standard. The bill also appropriately focuses on large companies developing the most powerful AI systems, while providing exemptions for smaller companies that are less likely to develop powerful models and should not bear unnecessary regulatory burdens. Of course, no major piece of legislation like SB 53 is perfect, nor do we expect it to be. But what is clear is that SB 53’s transparency requirements will have an important impact on frontier AI safety. Without it, labs with increasingly powerful models could face growing incentives to dial back their own safety and disclosure programs in order to compete. But with SB 53, developers can compete while ensuring they remain transparent about AI capabilities that pose risks to public safety, creating a level playing field.
The question before us all isn't whether we need AI governance—it's whether we'll develop it thoughtfully today or reactively tomorrow. SB 53 offers a solid path toward the former. We commend Senator Wiener and Governor Newsom for their leadership on responsible frontier AI governance, and we encourage the California Legislature to pass SB 53.
ARGUMENTS IN OPPOSITION: In an oppose-unless-amended position, CalChamber, Computer & Communications Industry Association, and TechNet jointly write:
[. . .]
We share your goal of ensuring the safe and responsible development of AI and appreciate efforts made in recent amendments to find common ground on how California should approach artificial intelligence models and we appreciate improvements made to the bill over the last several weeks. That being said, there are some issues of concern that remain and wish to flag certain other areas where the bill could be better aligned with the final findings of Governor Newsom’s Joint California Policy Working Group on AI Frontier Models, which arose out of his veto of SB 1047 (2024).
SB 53 should focus on model risk, not developer size—to fully address concerns about powerful models capable of catastrophic risk
We are concerned about the bill’s focus on “large developers” to the exclusion of other developers of models with advanced capabilities that pose risks of catastrophic harm. As amended September 5th, SB 53 now focuses on models that have a computational threshold of 10^26 floating point operations (or “FLOPs”) but only if those models are developed by entities with at least $500m in annual revenues.
Consistent with our position in SB 1047, we maintain that small entities can develop hugely influential and potentially risky models with similar capabilities to the models developed by “large developers”, as demonstrated by the Chinese company DeepSeek. As noted above, upon vetoing SB 1047, the Governor commissioned experts in the field to form the Joint California Working Group on AI Frontier Models, which has validated such concerns in their Final Reports, finding that small companies may create powerful models that pose safety risks. By excluding such models here, the bill fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones. The Governor’s Joint California Policy Working Group on AI Frontier Models cautions against developer-level thresholds stating:
Generic developer-level thresholds seem to be generally undesirable given the current AI landscape. Since many small entities can develop hugely influential and potentially risky foundation models, as demonstrated by the Chinese company DeepSeek, the use of thresholds based on developer-level properties may inadvertently ignore key players. […] At the same time, these approaches may bring into scope massive, established companies in other industries that are simply exploring the use of AI since thresholds based on properties of companies may not distinguish between the entire business and the AI-specific subset. Therefore, we caution against the use of customary developer-level metrics that do not consider the specifics of the AI industry and its associated technology.[42]
SB 53 should make clear that the AI ecosystem includes multiple actors including downstream developers
SB 53 does not account for the complexity of the AI value chain. Models are routinely adapted and fine-tuned by downstream developers in ways that could potentially increase risk. The bill should make clear that a frontier developer’s obligations do not extend to models that have been substantially modified by unaffiliated parties, otherwise accountability will be muddled and innovation chilled. We note that whereas the Governor’s Work Group report recognized the full AI ecosystem value chain, SB 53 still needs to fully recognize the roles of not just the original developer of a foundational model but also of those unaffiliated third parties who may modify and/or build on top of a foundation model. The bill should clarify these provisions to reflect the realities of the ecosystem, including downstream developers and open-source models.
SB 53 still raises concerns about protecting trade secrets and sensitive information, including matters of cybersecurity and national security.
We appreciate that amendments were made to change the level of detail required of the AI Safety Framework and changing summaries for transparency reports. However, SB 53 now requires a large developer only to transmit to the California Office of Emergency Services (CalOES) a summary of any assessment of catastrophic risk resulting from internal use of its frontier models every three months. Not only is this cadence of reporting unnecessary, CalOES will need to take serious steps to protect this information from being accessed by cybercriminals, foreign adversaries, or bad actors. Without ironclad safeguards, these transparency requirements could unintentionally make us less safe. The Joint California Policy Working Group on AI Frontier Models warns against this level of disclosure.
General details about risks of foundation models can be made public without undermining security, especially if these risks have been demonstrated in other foundation models or AI technologies. Specific details about concrete vulnerabilities should be disclosed carefully, with advanced notice to actors in the supply chain who are able to remediate them prior to broader disclosure.[43]
Requiring developers to justify redactions is less effective than not requiring developers to disclose any information that would include trade secrets, cybersecurity information, or other confidential or proprietary information.
SB 53 unnecessarily re-writes California Whistleblower law for just one industry
As amended, SB 53 rewrites California’s already robust whistleblower protections for just one industry. Creating a special, one-off standard for a single sector not only sets a poor precedent but also risks confusion and inconsistency across industries. Current law covers whistleblowing activities associated with AI safety because there is a robust body of existing law that governs whistleblower protection covering employees who report violations of state/federal laws, rules, or regulations. These laws are intentionally tied to actions that are illegal so there are clear lines of what is considered applicable and understood who gets protection when reporting. These protections cover activities associated with AI without creating unnecessary and confusing new processes in state law.
For example, Labor Code Section 6310 already protects whistleblowers who report unsafe working conditions or work practices. Similarly, federal laws such as the Sarbanes-Oxley Act protect employees who report safety violations or substantial and specific dangers to public health or safety. A brightline threshold is needed for what activity is covered so it is clear when a developer’s activities should be reported. For instance, in the field of research and development, innovations are being experimented with in novel contexts where there may be significant disagreement on what actions constitute risk. Thus, the bill mandates that there be an allegation of “specific and substantial danger to public health or safety resulting from a catastrophic risk,” the inherently subjective nature of these terms leaves room for differing interpretations as to what does or does not meet the threshold.
SB 53 requires steep penalties that are disproportionate for technical errors, inflexible incident reporting requirements, and no right to cure
As amended, SB 53 imposes a $1 million fine for a possible paperwork error which is excessive and risks punishing good-faith developers for technical mistakes rather than deterring real harm. Penalties should be fair, targeted, and proportionate. As we pointed out in our July 12th letter, SB 53 requires incident reporting within 15 days but does not provide flexibility for an investigation timeline. Even if 15 days is a reasonable reporting period, requirements should be flexible because all facts may not be known within 15 days of discovery. With respect to enforcement, we again state our view that the bill should grant businesses at least a 60 day right to cure, to ensure that law focuses on compliance and not punishment. In addition, given the highly detailed requirements of the bill as drafted, we think enforcement efforts should be focused on material failures to comply rather than also covering technical paperwork errors.
While we understand your focus on this issue and appreciate the recent amendments have made meaningful improvements to the prior version of the bill, given the immense promise of this technology, we believe that the bill would benefit from a focus on a risk-based framework for all frontier models, additional clarity in responsibilities among actors in the AI value chain, additional safeguards for trade secrets and security, and reasonable timelines, penalties, and enforcement provisions. [ . . .]
REGISTERED SUPPORT / OPPOSITION:
Support
Ai for Animals
Ai Futures Project
Ai Lab Watch
Ai Policy Tracker
All Girls Allowed
Anthropic Pbc
Apart Research
Association for Long Term Existence and Resilience (ALTER)
Berkeley Existential Risk Initiative (BERI)
California Federation of Labor Unions, Afl-cio
Center for Ai and Digital Policy
Center for Ai Policy
Center for Digital Democracy
Center for Human-compatible Ai
Center for Youth and Ai
Children's Advocacy Institute, University of San Diego School of Law
Common Sense Media
Depict.ai
Design It for US
District Council of Iron Workers of the State of California and Vicinity
Earningsstream LLC
Economic Security California Action
Elicit
Encode
Encode Ai Corporation
Encode Justice
Eon Systems
Existential Risk Observatory
Frontlines Foundation
Future of Life Institute
Indivisible California Statestrong
InnovateEDU
Momentum
Mothers Against Media Addiction
Nonlinear
Noso November
Oakland Privacy
Parents Television and Media Council
Parents Together Action
Public Interest Privacy Center
Redwood Research
Rights4girls
Scorecard
Secure Ai Future
Secure Ai Project
SEIU Califonia
Tech Oversight California
Techequity Action
The Brandes Lab At Nyu
The Midas Project
The Signals Network
Transparency Coalition.ai
Trevi Digital Assets Fund
University of California
Young People's Alliance
Youth Power Project
Opposition
Business Software Association
Chamber of Progress
Consumer Technology Association
Los Angeles County Business Federation (BIZFED) (UNREG)
Silicon Valley Leadership Group
Oppose Unless Amended
California Chamber of Commerce
Computer and Communications Industry Association
Insights Association
Technet
Analysis Prepared by: John Bennett / P. & C.P. / (916) 319-2200
For a discussion of issues with defining frontier models, see “California Report on Frontier AI Policy” (June 17, 2025), pp. 36-40, https://www.cafrontieraigov.org/. ↑
Pablo Arredondo, “GPT-4 Passes the Bar Exam: What That Means for Artificial Intelligence Tools in the Legal Profession” (Apr. 19, 2023), https://law.stanford.edu/2023/04/19/gpt-4-passes-the-bar-exam-what-that-means-for-artificial-intelligence-tools-in-the-legal-industry/. ↑
“International AI Safety Report,” AI Action Summit (Jan. 2025), p. 38, https://assets.publishing.service.gov.uk/media/679a0c48a77d250007d313ee/International_AI_Safety_Report_2025_accessible_f.pdf. ↑
Id. at p. 44. ↑
Id. at pp. 16-17. ↑
Id. at p. 27 ↑
Introducing OpenAI o3 and o4-mini OpenAI (Apr. 16, 2025), https://openai.com/index/introducing-o3-and-o4-mini/. ↑
Sam Altman, The Intelligence Age (Sept. 23, 2024), https://ia.samaltman.com/. ↑
Kyungtae Kim, “What is AGI, and when will it arrive?: Big Tech CEO Predictions” (Mar. 20, 2025), https://www.giz.ai/what-is-agi-and-when-will-it-arrive/; see also Kokotajlo et al, “AI 2027,” (Apr. 3, 2025), https://ai-2027.com/. ↑
John Markoff, “The Coming Superbrain,” New York Times (May 23, 2009), www.nytimes.com/2009/05/24/weekinreview/24markoff.html. ↑
Cade Metz, “Why We’re Unlikely to Get Artificial General Intelligence Anytime Soon,” New York Times (May 16, 2025), https://www.nytimes.com/2025/05/16/technology/what-is-agi.html. ↑
International AI Safety Report, supra, at pp. 17-21. The report does not address Lethal Autonomous Weapon Systems, which are typically narrow AI systems specifically developed for that purpose. (See id. at pp. 26-27.) ↑
Id. at p. 177 ↑
Cat Zakrzewski and Pranshu Verma, “New Hampshire opens criminal probe into AI calls impersonating Biden,” Washington Post, February 6, 2024, www.washingtonpost.com/technology/2024/02/06/nh-robocalls-ai-biden/. ↑
Emma Woollacott, “Yes, The Bots Really Are Taking Over The Internet”, Forbes (Apr. 16, 2024). https://www.forbes.com/sites/emmawoollacott/2024/04/16/yes-the-bots-really-are-taking-over-the-internet/. ↑
F. Salvi, M. H. Ribeiro, R. Gallotti, R. West, “On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial,” arXiv [cs.CY] (2024); http://arxiv.org/abs/2403.14380. ↑
Sharma et al, “Towards Understanding Sycophancy in Language Models” Arxiv (2023), https://arxiv.org/abs/2310.13548. ↑
Meghan Bobrowsky, “Zuckerberg’s Grand Vision: Most of Your Friends Will Be AI,” Wall Street Journal (May 7, 2025), https://www.wsj.com/tech/ai/mark-zuckerberg-ai-digital-future-0bb04de7?msockid=396cc204796e68e336e7d64978db69ac. ↑
Angela Yang, “Researchers secretly infiltrated a popular Reddit forum with AI bots, causing outrage,” NBC News (Apr. 29, 2025), https://www.nbcnews.com/tech/tech-news/reddiit-researchers-ai-bots-rcna203597. ↑
Introduction to AI Safety, Ethics, and Society, supra, at p. 11. ↑
International AI Safety Report, supra, at p. 72. ↑
Soice et al, “Can large language models democratize access to dual-use biotechnology?”
https://arxiv.org/pdf/2306.03809. To mitigate these risks, the authors propose measures such as third-party evaluations of LLMs before their release, curating training datasets to exclude harmful content, and implementing stringent screening of DNA synthesis orders. ↑
Fabio Urbina et al. “Dual use of artificial-intelligence-powered drug discovery”. In: Nature Machine Intelligence 4 (2022), pp. 189–191. ↑
Mummert et al., “What is reinforcement learning?” IBM Developer (September 15, 2022), https://developer.ibm.com/learningpaths/get-started-automated-ai-for-decision-making-api/what-is-automated-ai-for-decision-making./. ↑
Brian Christian, The Alignment Problem: Machine Learning and Human Values (Norton 2020, 1st ed.), pp. 9-11. ↑
International AI Safety Report, supra, at pp. 100-107. ↑
CAPTCHA is an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart.” ↑
OpenAI, “GPT-4 System Card,” https://cdn.openai.com/papers/gpt-4-system-card.pdf. ↑
Meinke et al, “Frontier Models are Capable of In-Context Scheming,” arXiv (Jan. 2025), https://arxiv.org/pdf/2412.04984. ↑
Harry Booth, “When AI Thinks It Will Lose, It Sometimes Cheat, Study Finds,” Time (Feb. 19, 2025), https://time.com/7259395/ai-chess-cheating-palisade-research/. ↑
System Card: Claude Opus 4 & Claude Sonnet 4, pp. 27, https://www.anthropic.com/claude-4-system-card. ↑
Center for AI Safety, “Statement on AI Risk: AI Experts and Public Figures Express Their Concern about AI Risk” (2024), https://www.safe.ai/work/statement-on-ai-risk. ↑
Id. at pp. 123-126. ↑
Dan Hendryks, Introduction to AI Safety, Ethics, and Society, p. 12, https://drive.google.com/file/d/1uph559W-ASR4MEn6M_7Mb3lqQTapC_gZ/view?pli=1. ↑
Bommasani,and Singer et al.. “The California Report on Frontier AI Policy.” The Joint California Policy Working Group on AI Frontier Models. June 17,2025. p 39. ↑
Id. at p. 26. ↑
Id. at p. 19. ↑
Id. at p. 21. ↑
Id.at p. 35. ↑
Id.at p. 29. ↑
RAISE Act can be found at https://www.nysenate.gov/legislation/bills/2025/A6453/amendment/A. ↑
Final Report at p. ↑
Id. at 30. ↑
SENATE THIRD READING
STR Bill Id:SB 53
Author:(Wiener)
As Amended Ver:September 5, 2025
Majority vote
SUMMARY
Requires developers of the most advanced, costly artificial intelligence (AI) systems to implement certain protocols and publically disclose the protocols they use to mitigate the risk of catastrophic harms. Requires the Office of Emergency Management to establish a mechanism by which developers and the public can report critical safety incidents. Provides for whistleblower protections and enforcement by the Attorney General (AG). Requires the Department of Technology to offer guidance to the Legislature on refining the scope over time to reflect technological advances. Provides, upon appropriation, for the creation of a framework to create a public cloud computing cluster.
Major Provisions
1) Requires large frontier developer who train AI models on a specified amount of compute and with revenues above $500 million to write, implement, comply with, and clearly and conspicuously publish on its internet website a frontier AI framework which details how the developer will address catastrophic harms and the protocols in place to address the materialization of such harms.
2) Requires a large frontier developer, before or at the time of making a new foundation model available, to publish on their internet website a transparency report for the model that describes the risk assessments or risk mitigation assessments used by the developer during the development of the model, including whether the developer used third-parties in the assessments and whether any denoted risk thresholds were attained. Developers who only reach the compute threshold must publish a higher level summary of the transparency report.
3) Requires the Office of Emergency Services to establish a mechanism for frontier model developers and the public to report critical safety incidents that have materialized as a result of the use of their models.
4) Requires the Department of Technology to offer guidance to the Legislature on redefining the scope of this bill beginning in 2027, which reflects the technological developments, scientific literature, national and international standards, as well as stakeholder engagement.
5) Upon appropriation, establishes in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as ″CalCompute″ that advances the development and deployment of AI, as prescribed.
6) Prohibits a developer from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliating against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has the authority to investigate the issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The developer′s activities pose a critical risk.
- The developer has made false or misleading statements about its management of critical risk.
COMMENTS
Foundation models, SB1047, and what this bill does. ″Frontier″ or ″foundation″ models are the largest and most powerful AI systems in development. These models are highly generalizable and have the potential to drive breakthroughs in science and medicine, streamline complex processes, and strengthen the economy. However, they also pose serious catastrophic risks due to their immense capabilities. A frontier model could help cure disease or design the next pandemic. It could automate bureaucratic functions or become autonomous and disrupt critical infrastructure.
Critics of AI safety argue that the evidence base is still too limited to justify regulation, and that prematurely imposing safeguards could stifle innovation. Responding to this dilemma, the International AI Safety Report, led by one of the ″godfathers of AI,″ Yoshua Bengio, has warned:
On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.
In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models by establishing a regulatory framework intended to prevent catastrophic harms. The bill would have required frontier model developers to create and implement comprehensive safety and security protocols before initiating training, to implement shutdown capabilities, and to perform risk assessments on models and implement reasonable safeguards, subject to third-party auditing, before releasing the models. The bill also would have prohibited releasing or using models that pose an unreasonable risk of catastrophic harms. Finally, SB 1047 would have created a new state agency – the Board of Frontier Models – to oversee the development of these models.
In vetoing the bill, Governor Gavin Newsom acknowledged the risks but emphasized the importance of addressing the evidence dilemma:
Let me be clear—I agree with the author—we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
Following the veto, the Governor convened the Joint California Policy Working Group on AI Frontier Models, which used empirical research, historical case studies, and modeling to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.
This bill seeks to implement the recommendations of the Working Group Report. Narrower than its predecessor, SB 53 takes an extremely light-touch approach that focuses on transparency as the means of ensuring safety and accountability for frontier model developers of the most powerful models. Large frontier developers who harness an extraordinarily high amount of compute power and have a revenue threshold over $500 million must create, implement, and publish both a safety and security protocol and a transparency report for each released model, whereas developers who only reach the compute threshold must only publish a high level transparency report. The bill does not prescribe any particular standards for these plans: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 50 deaths or $1 billion in damage.
The bill also establishes a critical incident reporting mechanism, administered by the Office of Emergency Management (OES), to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incidents must be reported within 15 days. The Department of Technology is further authorized to offer guidance to the legislature to redefine the scope of this bill to ensure that the bill remains responsive to technological advancements. Additionally, the bill establishes a consortium within the Government Operations Agency to create a public computing cluster, known as CalCompute, to support AI research and safety testing. The bill also provides whistleblower protections for employees of frontier model developers who report risks or noncompliance.
Furthermore, the bill incorporates additional recommendations from the Working Group Report. First, the bill requires developers to include capability thresholds in both their safety and security protocols and transparency reports. Developers must disclose when their models exceed these thresholds and must document any mitigation measures taken in response. Second, if a critical incident presents an imminent threat, the bill requires the developer to report the incident within 24 hours to the appropriate law enforcement authority. Third, the bill directs the OES and the Attorney General to publish an annual, anonymized, and aggregated summary of critical incidents and whistleblower reports. Ultimately, this would establish a transparency regime for the developers of frontier models.
For a full analysis please see the policy committee analysis.
According to the Author
In 2024, as part of his veto of Senate Bill 1047 (Wiener), Governor Newsom′s Joint California Working Group on AI Frontier Models was established – a group of top experts tasked with charting a course forward on AI policy for the developers of the most advanced AI systems. Their final report, released in June 2025, emphasized the growing evidence for risk of severe harm, such as ″AI-enabled hacking or biological attacks, and loss of control″ and argued ″California has a unique opportunity to continue supporting developments in frontier AI while addressing substantial risks that could have far-reaching consequences for the state and beyond.″
Drawing recommendations from Governor Newsom′s working group report, Senate Bill 53 requires covered developers to write, implement, and publish their safety and security protocol in redacted form to protect intellectual property. It would also require covered developers to report certain, carefully defined critical safety incidents to the Attorney General and would allow members of the public to report incidents.
SB 53 only applies to AI companies that have trained a model with 1026 floating point operations (FLOPs), a measure of computational power. These companies are spending hundreds of millions of dollars to train the most advanced AI models. As recommended by the Report, SB 53 also authorizes the Attorney General to adjust the scoping of the bill in the future to keep up with technological developments, but only focuses on well-resourced AI companies at the frontier of AI development.
Senate Bill 53 strengthens whistleblower protections for employees of frontier artificial intelligence laboratory companies whose activities pose a catastrophic risk. SB 53 also establishes a consortium to help create CalCompute: a public AI research cluster that will provide startups and researchers with access to the resources needed to develop large-scale AI systems.
In doing this, and building on Governor Newson′s working group report, SB 53 allows California to continue to maintain its leadership in the AI development ecosystem and to demonstrate that safety does not stifle success.
Arguments in Support
Secure AI Project, co-sponsors of the bill, alongside a coalition of technology equity advocacy groups write in support:
The California Report on Frontier AI Policy, while it does not endorse any specific legislation, forms the foundation for SB 53. Established by Governor Newsom in 2024 and led by Dr. Fei-Fei Li, Dr. Jennifer Tour Chayes and Mariano-Florentino Cuéllar, the report is anchored on the notion of ″trust but verify″ and calls for more transparency into the safety practices of AI companies, adverse event reporting requirements, and whistleblower protections. SB 53 implements these principles.
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The Report stated that ″some risks have unclear but growing evidence...AI-enabled hacking or biological attacks, and loss of control″ – the risks that SB 53 aims to address and gather more evidence about. Advanced AI is currently mostly unregulated, and these risks are currently being managed by companies themselves without any requirement that they inform the public about their risk management practices or report serious incidents. SB 53 addresses this much needed gap by implementing four key recommendations from the report.
First, the Report argued that ″transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk.″ SB 53 implements this recommendation as a requirement for large AI developers to write, publish, and follow safety and security protocols to manage the most severe risks. This is in line with voluntary commitments that companies have already made. Rather than prescribe specific technical standards that companies must take, the bill simply requires companies to be transparent about the approaches they are using. Some of the specific required elements of safety protocols, such as a requirement to manage risks related to internal use of AI models and cybersecurity policies, directly mirror recommendations in the Report. Others mirror components of the Stanford Foundation Model Transparency Index, which is cited prominently in the Report.
Second, the Report stated that ″transparency into pre-deployment assessments of capabilities and risks, spanning both developer-conducted and externally conducted evaluations, is vital given that these evaluations are early indicators of how models may affect society and may be interpreted (potentially undesirably) as safety assurances.″ SB 53 accomplishes this with a requirement that large developers publish transparency reports that include the results of their pre-deployment assessments of catastrophic risk. The Report also argues that ″transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety,″ which forms the basis for 22757.12(c)(3).
Third, the Report concluded that ″an adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base.″ SB 53 takes exactly this approach by establishing a tightly defined set of critical safety incidents that AI developers are required to report to the Attorney General. It would also allow members of the public to optionally submit reports.
Finally, the Report recommends strengthening whistleblower protections, pointing out that ″actions that may clearly pose a risk and violate company policies...may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of activities, which may draw upon notions of ′good faith′ reporting on risks found in other domains such as cybersecurity.″ This recommendation is mirrored in SB 53, which allows employees to report evidence of catastrophic risks as well as violations of SB 53 itself to government authorities with legal protections against retaliation.
SB 53 only applies to the largest AI developers – those training models with more than 10^26 floating point operations (FLOPs). These are companies spending hundreds of millions or billions of dollars to train the most advanced AI models. It would impose no burden on smaller companies and the requirements it imposes on large companies are minimal compared to what companies are already voluntarily doing. The Report argues that ″policymakers should ensure that mechanisms are in place to adapt thresholds over time—not only by updating specific threshold values but also by revising or replacing metrics if needed.″ It also suggests specific criteria that thresholds should be evaluated for. Following this recommendation, SB 53 allows the Attorney General to update the definition of ″large developer″ through regulation while considering the same factors described in the report. Regardless of any update, the Attorney General must only include ″well-resourced large developers at the frontier of artificial intelligence development″ in the scoping of the bill. If legislation is needed to cover other developers, the Attorney General is instructed to write a report to the Legislature requesting it.
Finally, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California′s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. This mirrors a similar computing cluster that is already being established in New York state. We support this groundbreaking effort, which would advance and democratize AI research in California.
SB 53 thoughtfully implements the recommendations of the Report by combining a low-burden transparency and reporting regime with a public compute cluster that will broaden access for AI researchers and startups in California. This is a commonsense approach that will strengthen the AI ecosystem, benefiting both companies and the public interest.
For all these reasons, we respectfully urge your support of this important measure.
Arguments in Opposition
In opposition to the bill, Chamber of Progress argues:
On behalf of the Chamber of Progress, a tech industry association supporting public policies to build a more inclusive society in which all people benefit from technological advances, we respectfully urge you to oppose SB 53, based on its recent amendments.
The definition of ″catastrophic risk″ remains vague and overreaching
While the amended bill replaces the term ″critical risk″ with ″catastrophic risk,″ the underlying problem persists. The definition remains overly expansive and ambiguous, capturing a wide array of hypothetical scenarios that may not reflect real-world AI capabilities or threats.
Under Section 22757.11(b), the definition of ″catastrophic risk″ includes scenarios where a foundation model is ″materially likely″ to cause harm, potentially due to misuse or malicious inputs. However, this standard is vague, lacks clear and objective thresholds, and leaves room for subjective interpretation by whistleblowers or regulators. In a rapidly evolving field like AI, such ambiguity could unfairly penalize developers who are acting responsibly.
In addition, the inclusion of highly abstract risks, such as the evasion of human control under Section 22757.12(a)(2), creates significant uncertainty. Without clear technical criteria, companies may face liability or investigation based on assumptions about what a model might enable rather than what it has demonstrably done. This uncertainty undermines research and commercial deployment in California and could push critical AI development efforts out of state or abroad.
The $100,000,000 compute cost threshold risks misidentifying frontier AI models
SB 53′s use of an arbitrary $100,000,000 compute cost threshold to determine eligibility for protections is an inherently flawed method for identifying frontier AI models. This threshold may result in the overinclusion of developers working on benign systems while potentially excluding smaller models that pose significant real-world risks. It also ignores the constantly changing cost of compute.
A more effective approach would involve a threshold based on model capabilities, deployment context, and specific use cases rather than relying solely on computational costs.
SB 53′s extensive safety and security protocols create impractical burdens for AI developers
SB 53 imposes comprehensive safety and security requirements on AI developers, as outlined in Section 22757.12(a), including risk testing, deployment practices, and escalation procedures. While these objectives are important, the bill demands an impractical level of detailed planning and documentation for every conceivable misuse scenario, many of which are speculative or unrealistic.
This exhaustive approach compels developers to allocate significant time and resources toward preparing for hypothetical risks rather than addressing actual, demonstrable harms. For startups and smaller companies, these extensive protocols create a heavy administrative burden that diverts critical resources away from innovation and the timely deployment of beneficial AI technologies.
Additionally, Section 22757.12(c)′s requirement that developers publish detailed transparency reports, before or at the time of deploying a new or substantially modified foundation model, creates significant risks to both competitiveness and operational security.
Although redactions are permitted under subsection (f), the requirement to publish the ″character and justification″ of redacted material could still inadvertently expose business-sensitive strategies or vulnerabilities. This level of forced transparency goes beyond reasonable accountability and may discourage responsible companies from operating in California. It also creates opportunities for misuse by malicious actors who could exploit disclosed model weaknesses or mitigation gaps.
In fast-moving AI markets, publication of this level of detail erodes a developer′s ability to maintain a competitive edge and deters innovation by raising legal and reputational risks associated with even speculative harms.
FISCAL COMMENTS
According to the Assembly Appropriations Committee:
- Costs (General Fund) to the Department of Justice (DOJ), likely in the low millions of dollars annually, to establish reporting mechanisms, review critical incident reports, conduct investigations, publish reports, and enforce violations. DOJ anticipates costs of approximately $1.1 million in fiscal year 2025-26 and $2 million annually ongoing thereafter for eight staff positions (attorneys, analysts, IT specialists, and legal secretaries) in its Consumer Protection Section and external consultant costs. DOJ reports it is unable to absorb these costs and can implement this bill only with an appropriation of additional funding. DOJ may also incur enforcement costs for violations of the bill′s whistleblower protections; the bill does not clearly specify the entity responsible for this enforcement but permits an employee making a whistleblower report to use an existing DOJ whistleblower hotline.
- Costs (General Fund) to GovOps to establish and operate the CalCompute consortium until January 1, 2027, possibly in the high hundreds of thousands of dollars to low millions of dollars. GovOps estimates total costs of $2.5 million for expert contractors, infrastructure planning, and staffing to manage the project, conduct research, and develop the required report. GovOps was not able to provide a breakdown of these costs but anticipates the workload would be handled by contract workers due to the short timeframe in the bill. Members of the consortium are not entitled to compensation but are entitled to reimbursement for necessary expenses incurred in performing their duties; these costs were not included in GovOps′ fiscal estimate but may be in the thousands to low tens of thousands of dollars depending on the activities of the consortium.
- Possible cost pressures (General Fund) of an unknown but potentially significant amount to the UC to operate CalCompute, should it be established within the UC. State costs may be offset to some extent by private donations, which the bill authorizes the UC to receive to implement CalCompute.
- Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce violations of the bill′s whistleblower provisions. Actual costs will depend on the number of violations, the number of actions pursued, and the amount of workload associated with each action.
- Cost pressures (Trial Court Trust Fund, General Fund) to the courts to adjudicate enforcement actions and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2025-26 state budget provides $82 million ongoing General Fund to the Trial Court Trust Fund for court operations.
VOTES
SENATE FLOOR: 37-0-3
YES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
ABS, ABST OR NV: Cervantes, Limón, Reyes
ASM JUDICIARY: 12-0-0
YES: Kalra, Dixon, Bauer-Kahan, Bryan, Connolly, Harabedian, Macedo, Pacheco, Papan, Sanchez, Stefani, Zbur
ASM PRIVACY AND CONSUMER PROTECTION: 10-0-5
YES: Bauer-Kahan, Dixon, Irwin, Lowenthal, McKinnor, Ortega, Pellerin, Petrie-Norris, Ward, Wilson
ABS, ABST OR NV: Bryan, DeMaio, Macedo, Patterson, Wicks
ASM APPROPRIATIONS: 11-1-3
YES: Wicks, Arambula, Calderon, Caloza, Elhawary, Fong, Mark González, Ahrens, Pacheco, Pellerin, Solache
NO: Tangipa
ABS, ABST OR NV: Sanchez, Dixon, Ta
UPDATED
VERSION: September 5, 2025
CONSULTANT: John Bennett / P. & C.P. / (916) 319-2200 FN: 0001794
SENATE THIRD READING
STR Bill Id:SB 53
Author:(Wiener)
As Amended Ver:September 02, 2025
Majority vote
SUMMARY
Requires large developers of the most advanced, costly artificial intelligence (AI) systems to implement certain protocols and publically disclose the protocols they use to mitigate the risk of catastrophic harms. Requires operators of computing clusters to obtain specified information relating to customers. Provides for whistleblower protections and enforcement by the Attorney General (AG). Provides, upon appropriation, for the creation of a framework to create a public cloud computing cluster.
Major Provisions
1) Requires large developers who train AI models on a specified amount of compute and with revenues above $500 million to write, implement, comply with, and clearly and conspicuously publish on its internet website a safety and security protocol which details how the developer will address catastrophic harms and the protocols in place to address the materialization of such harms.
2) Requires a large developer, before or at the time of making a new foundation model available, to publish on their internet website a transparency report for the model that describes the risk assessments or risk mitigation assessments used by the developer during the development of the model, including whether the developer used third-parties in the assessments and whether any denoted risk thresholds were attained.
3) Requires the Attorney General to establish a mechanism for large developers to report critical safety incidents that have materialized as a result of the use of their models.
4) Permits the Attorney General to redefine the definition of a large developer beginning in 2027, which reflects the technological developments, scientific literature, national and international standards, as well as stakeholder engagement.
5) Upon appropriation, establishes in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as "CalCompute" that advances the development and deployment of AI, as prescribed.
6) Prohibits a developer from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliating against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has the authority to investigate the issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The developer's activities pose a critical risk.
- The developer has made false or misleading statements about its management of critical risk.
COMMENTS
Foundation models, SB1047, and what this bill does. "Frontier" or "foundation" models are the largest and most powerful AI systems in development. These models are highly generalizable and have the potential to drive breakthroughs in science and medicine, streamline complex processes, and strengthen the economy. However, they also pose serious catastrophic risks due to their immense capabilities. A frontier model could help cure disease or design the next pandemic. It could automate bureaucratic functions or become autonomous and disrupt critical infrastructure.
Critics of AI safety argue that the evidence base is still too limited to justify regulation, and that prematurely imposing safeguards could stifle innovation. Responding to this dilemma, the International AI Safety Report, led by one of the "godfathers of AI," Yoshua Bengio, has warned:
On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.
In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models by establishing a regulatory framework intended to prevent catastrophic harms. The bill would have required frontier model developers to create and implement comprehensive safety and security protocols before initiating training, to implement shutdown capabilities, and to perform risk assessments on models and implement reasonable safeguards, subject to third-party auditing, before releasing the models. The bill also would have prohibited releasing or using models that pose an unreasonable risk of catastrophic harms. Finally, SB 1047 would have created a new state agency – the Board of Frontier Models – to oversee the development of these models.
In vetoing the bill, Governor Gavin Newsom acknowledged the risks but emphasized the importance of addressing the evidence dilemma:
Let me be clear—I agree with the author—we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
Following the veto, the Governor convened the Joint California Policy Working Group on AI Frontier Models, which used empirical research, historical case studies, and modeling to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.
This bill seeks to implement the recommendations of the Working Group Report. Narrower than its predecessor, SB 53 takes a light-touch approach that focuses on transparency as the means of ensuring safety and accountability for the largest developers of the most powerful models. Large developers who harness an extraordinarily high amount of compute power must create, implement, and publish both a safety and security protocol and a transparency report for each released model. The bill does not prescribe any particular standards for these plans: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 50 deaths or $1 billion in damage.
The bill also establishes a critical incident reporting mechanism, administered by the Attorney General (AG), to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incidents must be reported within 15 days. The AG is further authorized to update the definition of a "large developer" through rulemaking to ensure that the bill remains responsive to technological advancements. Additionally, the bill establishes a consortium within the Government Operations Agency to create a public computing cluster, known as CalCompute, to support AI research and safety testing. The bill also provides whistleblower protections for employees and contractors of large developers who report risks or noncompliance. The AG is authorized to enforce the bill by seeking an unspecified civil penalty.
Furthermore, the bill incorporates additional recommendations from the Working Group Report. First, the bill requires developers to include capability thresholds in both their safety and security protocols and transparency reports. Developers must disclose when their models exceed these thresholds and must document any mitigation measures taken in response. Second, to avoid over-burdening smaller startups, the amendments add to the definition of "large developer" an annual revenue threshold of $500 million, in addition to the current compute threshold. Third, the bill requires developers to disclose whether they retain the ability to shut down a model under their control in the event of a critical incident. This is a disclosure requirement only; unlike SB 1047, this bill does not mandate that a developer maintain the technical ability to shut down a model. Fourth, if a critical incident presents an imminent threat, the bill requires the developer to report the incident within 24 hours to the appropriate law enforcement authority. Fifth, the bill directs the AG to publish an annual, anonymized, and aggregated summary of all critical incident and whistleblower reports. Ultimately, this would establish a baseline transparency regime for the largest developers of foundation models.
For a full analysis please see the policy committee analysis.
According to the Author
In 2024, as part of his veto of Senate Bill 1047 (Wiener), Governor Newsom's Joint California Working Group on AI Frontier Models was established – a group of top experts tasked with charting a course forward on AI policy for the developers of the most advanced AI systems. Their final report, released in June 2025, emphasized the growing evidence for risk of severe harm, such as "AI-enabled hacking or biological attacks, and loss of control" and argued "California has a unique opportunity to continue supporting developments in frontier AI while addressing substantial risks that could have far-reaching consequences for the state and beyond."
Drawing recommendations from Governor Newsom's working group report, Senate Bill 53 requires covered developers to write, implement, and publish their safety and security protocol in redacted form to protect intellectual property. It would also require covered developers to report certain, carefully defined critical safety incidents to the Attorney General and would allow members of the public to report incidents.
SB 53 only applies to AI companies that have trained a model with 1026 floating point operations (FLOPs), a measure of computational power. These companies are spending hundreds of millions of dollars to train the most advanced AI models. As recommended by the Report, SB 53 also authorizes the Attorney General to adjust the scoping of the bill in the future to keep up with technological developments, but only focuses on well-resourced AI companies at the frontier of AI development.
Senate Bill 53 strengthens whistleblower protections for employees of frontier artificial intelligence laboratory companies whose activities pose a catastrophic risk. SB 53 also establishes a consortium to help create CalCompute: a public AI research cluster that will provide startups and researchers with access to the resources needed to develop large-scale AI systems.
In doing this, and building on Governor Newson's working group report, SB 53 allows California to continue to maintain its leadership in the AI development ecosystem and to demonstrate that safety does not stifle success.
Arguments in Support
Secure AI Project, co-sponsors of the bill, alongside a coalition of technology equity advocacy groups write in support:
The California Report on Frontier AI Policy, while it does not endorse any specific legislation, forms the foundation for SB 53. Established by Governor Newsom in 2024 and led by Dr. Fei-Fei Li, Dr. Jennifer Tour Chayes and Mariano-Florentino Cuéllar, the report is anchored on the notion of "trust but verify" and calls for more transparency into the safety practices of AI companies, adverse event reporting requirements, and whistleblower protections. SB 53 implements these principles.
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The Report stated that "some risks have unclear but growing evidence...AI-enabled hacking or biological attacks, and loss of control" – the risks that SB 53 aims to address and gather more evidence about. Advanced AI is currently mostly unregulated, and these risks are currently being managed by companies themselves without any requirement that they inform the public about their risk management practices or report serious incidents. SB 53 addresses this much needed gap by implementing four key recommendations from the report.
First, the Report argued that "transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk." SB 53 implements this recommendation as a requirement for large AI developers to write, publish, and follow safety and security protocols to manage the most severe risks. This is in line with voluntary commitments that companies have already made. Rather than prescribe specific technical standards that companies must take, the bill simply requires companies to be transparent about the approaches they are using. Some of the specific required elements of safety protocols, such as a requirement to manage risks related to internal use of AI models and cybersecurity policies, directly mirror recommendations in the Report. Others mirror components of the Stanford Foundation Model Transparency Index, which is cited prominently in the Report.
Second, the Report stated that "transparency into pre-deployment assessments of capabilities and risks, spanning both developer-conducted and externally conducted evaluations, is vital given that these evaluations are early indicators of how models may affect society and may be interpreted (potentially undesirably) as safety assurances." SB 53 accomplishes this with a requirement that large developers publish transparency reports that include the results of their pre-deployment assessments of catastrophic risk. The Report also argues that "transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety," which forms the basis for 22757.12(c)(3).
Third, the Report concluded that "an adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base." SB 53 takes exactly this approach by establishing a tightly defined set of critical safety incidents that AI developers are required to report to the Attorney General. It would also allow members of the public to optionally submit reports.
Finally, the Report recommends strengthening whistleblower protections, pointing out that "actions that may clearly pose a risk and violate company policies...may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of activities, which may draw upon notions of 'good faith' reporting on risks found in other domains such as cybersecurity." This recommendation is mirrored in SB 53, which allows employees to report evidence of catastrophic risks as well as violations of SB 53 itself to government authorities with legal protections against retaliation.
SB 53 only applies to the largest AI developers – those training models with more than 10^26 floating point operations (FLOPs). These are companies spending hundreds of millions or billions of dollars to train the most advanced AI models. It would impose no burden on smaller companies and the requirements it imposes on large companies are minimal compared to what companies are already voluntarily doing. The Report argues that "policymakers should ensure that mechanisms are in place to adapt thresholds over time—not only by updating specific threshold values but also by revising or replacing metrics if needed." It also suggests specific criteria that thresholds should be evaluated for. Following this recommendation, SB 53 allows the Attorney General to update the definition of "large developer" through regulation while considering the same factors described in the report. Regardless of any update, the Attorney General must only include "well-resourced large developers at the frontier of artificial intelligence development" in the scoping of the bill. If legislation is needed to cover other developers, the Attorney General is instructed to write a report to the Legislature requesting it.
Finally, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California's top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. This mirrors a similar computing cluster that is already being established in New York state. We support this groundbreaking effort, which would advance and democratize AI research in California.
SB 53 thoughtfully implements the recommendations of the Report by combining a low-burden transparency and reporting regime with a public compute cluster that will broaden access for AI researchers and startups in California. This is a commonsense approach that will strengthen the AI ecosystem, benefiting both companies and the public interest.
For all these reasons, we respectfully urge your support of this important measure.
Arguments in Opposition
In opposition to the bill, Chamber of Progress argues:
On behalf of the Chamber of Progress, a tech industry association supporting public policies to build a more inclusive society in which all people benefit from technological advances, we respectfully urge you to oppose SB 53, based on its recent amendments.
The definition of "catastrophic risk" remains vague and overreaching
While the amended bill replaces the term "critical risk" with "catastrophic risk," the underlying problem persists. The definition remains overly expansive and ambiguous, capturing a wide array of hypothetical scenarios that may not reflect real-world AI capabilities or threats.
Under Section 22757.11(b), the definition of "catastrophic risk" includes scenarios where a foundation model is "materially likely" to cause harm, potentially due to misuse or malicious inputs. However, this standard is vague, lacks clear and objective thresholds, and leaves room for subjective interpretation by whistleblowers or regulators. In a rapidly evolving field like AI, such ambiguity could unfairly penalize developers who are acting responsibly.
In addition, the inclusion of highly abstract risks, such as the evasion of human control under Section 22757.12(a)(2), creates significant uncertainty. Without clear technical criteria, companies may face liability or investigation based on assumptions about what a model might enable rather than what it has demonstrably done. This uncertainty undermines research and commercial deployment in California and could push critical AI development efforts out of state or abroad.
The $100,000,000 compute cost threshold risks misidentifying frontier AI models
SB 53's use of an arbitrary $100,000,000 compute cost threshold to determine eligibility for protections is an inherently flawed method for identifying frontier AI models. This threshold may result in the overinclusion of developers working on benign systems while potentially excluding smaller models that pose significant real-world risks. It also ignores the constantly changing cost of compute.
A more effective approach would involve a threshold based on model capabilities, deployment context, and specific use cases rather than relying solely on computational costs.
SB 53's extensive safety and security protocols create impractical burdens for AI developers
SB 53 imposes comprehensive safety and security requirements on AI developers, as outlined in Section 22757.12(a), including risk testing, deployment practices, and escalation procedures. While these objectives are important, the bill demands an impractical level of detailed planning and documentation for every conceivable misuse scenario, many of which are speculative or unrealistic.
This exhaustive approach compels developers to allocate significant time and resources toward preparing for hypothetical risks rather than addressing actual, demonstrable harms. For startups and smaller companies, these extensive protocols create a heavy administrative burden that diverts critical resources away from innovation and the timely deployment of beneficial AI technologies.
Additionally, Section 22757.12(c)'s requirement that developers publish detailed transparency reports, before or at the time of deploying a new or substantially modified foundation model, creates significant risks to both competitiveness and operational security.
Although redactions are permitted under subsection (f), the requirement to publish the "character and justification" of redacted material could still inadvertently expose business-sensitive strategies or vulnerabilities. This level of forced transparency goes beyond reasonable accountability and may discourage responsible companies from operating in California. It also creates opportunities for misuse by malicious actors who could exploit disclosed model weaknesses or mitigation gaps.
In fast-moving AI markets, publication of this level of detail erodes a developer's ability to maintain a competitive edge and deters innovation by raising legal and reputational risks associated with even speculative harms.
FISCAL COMMENTS
According to the Assembly Appropriations Committee:
- Costs (General Fund) to the Department of Justice (DOJ), likely in the low millions of dollars annually, to establish reporting mechanisms, review critical incident reports, conduct investigations, publish reports, and enforce violations. DOJ anticipates costs of approximately $1.1 million in fiscal year 2025-26 and $2 million annually ongoing thereafter for eight staff positions (attorneys, analysts, IT specialists, and legal secretaries) in its Consumer Protection Section and external consultant costs. DOJ reports it is unable to absorb these costs and can implement this bill only with an appropriation of additional funding. DOJ may also incur enforcement costs for violations of the bill's whistleblower protections; the bill does not clearly specify the entity responsible for this enforcement but permits an employee making a whistleblower report to use an existing DOJ whistleblower hotline.
- Costs (General Fund) to GovOps to establish and operate the CalCompute consortium until January 1, 2027, possibly in the high hundreds of thousands of dollars to low millions of dollars. GovOps estimates total costs of $2.5 million for expert contractors, infrastructure planning, and staffing to manage the project, conduct research, and develop the required report. GovOps was not able to provide a breakdown of these costs but anticipates the workload would be handled by contract workers due to the short timeframe in the bill. Members of the consortium are not entitled to compensation but are entitled to reimbursement for necessary expenses incurred in performing their duties; these costs were not included in GovOps' fiscal estimate but may be in the thousands to low tens of thousands of dollars depending on the activities of the consortium.
- Possible cost pressures (General Fund) of an unknown but potentially significant amount to the UC to operate CalCompute, should it be established within the UC. State costs may be offset to some extent by private donations, which the bill authorizes the UC to receive to implement CalCompute.
- Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce violations of the bill's whistleblower provisions. Actual costs will depend on the number of violations, the number of actions pursued, and the amount of workload associated with each action.
- Cost pressures (Trial Court Trust Fund, General Fund) to the courts to adjudicate enforcement actions and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2025-26 state budget provides $82 million ongoing General Fund to the Trial Court Trust Fund for court operations.
VOTES
SENATE FLOOR: 37-0-3
YES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
ABS, ABST OR NV: Cervantes, Limón, Reyes
ASM JUDICIARY: 12-0-0
YES: Kalra, Dixon, Bauer-Kahan, Bryan, Connolly, Harabedian, Macedo, Pacheco, Papan, Sanchez, Stefani, Zbur
ASM PRIVACY AND CONSUMER PROTECTION: 10-0-5
YES: Bauer-Kahan, Dixon, Irwin, Lowenthal, McKinnor, Ortega, Pellerin, Petrie-Norris, Ward, Wilson
ABS, ABST OR NV: Bryan, DeMaio, Macedo, Patterson, Wicks
ASM APPROPRIATIONS: 11-1-3
YES: Wicks, Arambula, Calderon, Caloza, Elhawary, Fong, Mark González, Ahrens, Pacheco, Pellerin, Solache
NO: Tangipa
ABS, ABST OR NV: Sanchez, Dixon, Ta
UPDATED
VERSION: September 02, 2025
CONSULTANT: John Bennett / P. & C.P. / (916) 319-2200 FN: 0001505
Date of Hearing: August 20, 2025
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
SBPCA Bill Id:SB 53
Author:(Wiener) – As Amended Ver:July 17, 2025
Policy Committee: | Judiciary | Vote: | 12 - 0 |
Privacy and Consumer Protection | 10 - 0 | ||
|
|
Urgency: No State Mandated Local Program: No Reimbursable: No
SUMMARY:
This bill imposes reporting and auditing requirements on large developers of foundation artificial intelligence (AI) models (“foundation models’), requires the Government Operations Agency (GovOps) to develop a framework for a public cloud computing cluster, and enacts whistleblower protections related to activities of large developers.
Specifically, among other provisions, this bill:
- Requires a large developer to implement, publish, and update a safety and security protocol that includes, among other specified elements, how the developer assesses catastrophic risks from a foundation model, the actions the developer will take if a risk threshold is attained by a model, the role of third-party assessment of catastrophic risk in the developer’s protocol, and the developer’s cybersecurity practices.
- Requires a large developer to publish a transparency report before or at the same time as the developer deploys a new foundation model. The report must include specified information about any risk assessment performed on the new model and the developer’s reasoning behind deploying the new model.
- Requires a large developer to publish any assessment of catastrophic risk or dangerous capabilities resulting from internal use of its foundation models, according to the schedule the developer specifies in its safety and security protocol.
- Prohibits a large developer from making a materially false or misleading statement about catastrophic risk from its foundation models, its management of catastrophic risk, or its implementation of or compliance with its safety and security protocol.
- Requires the Attorney General (AG) to establish a mechanism through which a large developer or a member of the public may report a critical safety incident, and requires critical safety incident reporting by a large developer, as specified.
- Requires the AG to review critical safety incidents submitted by a large developer and permits the AG to review critical safety incidents submitted by members of the public.
- Requires the AG, beginning January 1, 2027, to produce an annual report of anonymized and aggregated information from critical safety incident reports, whistleblower reports, and summaries of auditor’s reports. The report must be distributed to the Governor’s Office and the Legislature.
- Requires a large developer, beginning January 1, 2030, and at least annually thereafter, to retain an independent third party auditor to produce a report assessing whether the developer has substantially complied with its safety and security protocol, as specified, and requires an auditor to transmit to the AG a high-level summary of the audit report within 30 days after completing an audit.
- Authorizes the AG to adopt regulations to update the definition of a “large developer” to ensure it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of AI development.
- Authorizes the AG to bring a civil action to enforce a violation of the bill’s provisions by a large developer and imposes civil penalties ranging from up to $10,000 to $10 million depending on the risk of harm associated with the violation and whether the violation was unknowing or knowing.
- Authorizes the AG to bring a civil action to enforce a violation of the bill’s provisions by an auditor and imposes a civil penalty of up to $10,000 for each violation.
- Establishes within GovOps a consortium, as specified, to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute,” and specifies elements that must be included in the framework, elements that must be included in CalCompute, and membership requirements for the consortium.
- Requires GovOps, on or before January 1, 2027, to submit a report from the consortium to the Legislature with the framework for the creation and operation of CalCompute, as specified. Dissolves the consortium following submission of the report to the Legislature.
- Requires the consortium to make reasonable efforts to ensure CalCompute is established within the University of California (UC) to the extent possible. If CalCompute is established within the UC, authorizes the UC to receive private donations for the purposes of implementing CalCompute.
- Makes the bill’s provisions pertaining to CalCompute operative only upon appropriation.
- Establishes whistleblower protections for people who interact with large developers, including employees and contractors, among others, if they disclose information they have a reasonable cause to believe indicates the developer’s activities pose a catastrophic risk or the developer has violated the bill’s provisions, and requires a large developer to post notices and provide internal processes to facilitate information reporting.
FISCAL EFFECT:
- Costs (General Fund) to the Department of Justice (DOJ), likely in the low millions of dollars annually, to establish reporting mechanisms, review critical incident reports, conduct investigations, publish reports, and enforce violations. DOJ anticipates costs of approximately $1.1 million in fiscal year 2025-26 and $2 million annually ongoing thereafter for eight staff positions (attorneys, analysts, IT specialists, and legal secretaries) in its Consumer Protection Section and external consultant costs. DOJ reports it is unable to absorb these costs and can implement this bill only with an appropriation of additional funding. DOJ may also incur enforcement costs for violations of the bill’s whistleblower protections; the bill does not clearly specify the entity responsible for this enforcement but permits an employee making a whistleblower report to use an existing DOJ whistleblower hotline.
- Costs (General Fund) to GovOps to establish and operate the CalCompute consortium until January 1, 2027, possibly in the high hundreds of thousands of dollars to low millions of dollars. GovOps estimates total costs of $2.5 million for expert contractors, infrastructure planning, and staffing to manage the project, conduct research, and develop the required report. GovOps was not able to provide a breakdown of these costs but anticipates the workload would be handled by contract workers due to the short timeframe in the bill. Members of the consortium are not entitled to compensation but are entitled to reimbursement for necessary expenses incurred in performing their duties; these costs were not included in GovOps’ fiscal estimate but may be in the thousands to low tens of thousands of dollars depending on the activities of the consortium.
- Possible cost pressures (General Fund) of an unknown but potentially significant amount to the UC to operate CalCompute, should it be established within the UC. State costs may be offset to some extent by private donations, which the bill authorizes the UC to receive to implement CalCompute.
- Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce violations of the bill’s whistleblower provisions. Actual costs will depend on the number of violations, the number of actions pursued, and the amount of workload associated with each action.
- Cost pressures (Trial Court Trust Fund, General Fund) to the courts to adjudicate enforcement actions and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2025-26 state budget provides $82 million ongoing General Fund to the Trial Court Trust Fund for court operations.
COMMENTS:
As the development of AI models has progressed, scholars, policymakers, philosophers, and others have raised concern about the lack of regulation for the most powerful emerging AI models. As discussed thoroughly in the policy committee analyses of this bill, many people inside and outside the industry have encouraged governments to put an anticipatory regulatory framework in place now, despite the considerable uncertainty about the future of AI modeling, to mitigate the risk of future harm resulting from AI models.
Last year, the Legislature passed SB 1047 (Wiener), which, among other provisions, would have established a state board to regulate frontier AI models and imposed testing and reporting requirements on developers before they could train, use, or make publicly available a covered frontier model. SB 1047 also contained some elements that were similar to this bill, including whistleblower protections and a consortium to create a public computing cluster. Governor Newsom vetoed SB 1047. In a lengthy veto message, the Governor cited the fact that the bill would have rigorously regulated only the most expensive and large-scale models, without taking into account the particular features and applications of each model:
While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.
On the same day he vetoed SB 1047, Governor Newsom announced he had convened a working group of experts to “help California develop workable guardrails” for generative AI and frontier AI models. In June 2025, the working group released its final report on frontier AI policy. The report includes discussion of and recommendations related to many aspects of AI development and regulation, including transparency into the activities of AI companies, how adverse events should be reported to regulators and the public, and factors to consider when tailoring the scope of state regulations on developers and deployers of AI technology.
This bill integrates recommendations of the working group, modified elements of SB 1047, and input from other stakeholders to create a regulatory regime for foundation AI models. According to the author, the result is a more balanced approach that “allows California to continue to maintain its leadership in the AI development ecosystem and to demonstrate that safety does not stifle success.” The bill is sponsored by Encode Justice, Secure AI Project, and Economic Security California Action, and is supported by groups that favor greater regulation of the AI industry. The bill is opposed by business and industry representatives, including the California Chamber of Commerce, Chamber of Progress, and TechNet.
Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081
Date of Hearing: July 16, 2025
Fiscal:
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
SBPCA Bill Id:SB 53
Author:(Wiener) – As Amended Ver:July 8, 2025
SENATE VOTE: 37-0
PROPOSED AMENDMENTS
SUBJECT: CalCompute: foundation models: whistleblowers
SYNOPSIS
“Frontier” or “foundation” models are the largest and most powerful AI systems in development. These models are highly generalizable and have the potential to drive breakthroughs in science and medicine, streamline complex processes, and strengthen the economy. However, they also pose serious catastrophic risks due to their immense capabilities. A frontier model could help cure disease or design the next pandemic. It could automate bureaucratic functions or become autonomous and disrupt critical infrastructure.
Critics of AI safety argue that the evidence base is still too limited to justify regulation, and that prematurely imposing safeguards could stifle innovation. Responding to this dilemma, the International AI Safety Report, led by one of the “godfathers of AI,” Yoshua Bengio, has warned:
On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.
In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models by establishing a regulatory framework intended to prevent catastrophic harms. The bill would have required frontier model developers to create and implement comprehensive safety and security protocols before initiating training, to implement shutdown capabilities, and to perform risk assessments on models and implement reasonable safeguards, subject to third-party auditing, before releasing the models. The bill also would have prohibited releasing or using models that pose an unreasonable risk of catastrophic harms. Finally, SB 1047 would have created a new state agency – the Board of Frontier Models – to oversee the development of these models.
In vetoing the bill, Governor Gavin Newsom acknowledged the risks but emphasized the importance of addressing the evidence dilemma:
Let me be clear—I agree with the author—we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
Following the veto, the Governor convened the Joint California Policy Working Group on AI Frontier Models, which used empirical research, historical case studies, and modeling to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.
This bill seeks to implement the recommendations of the Working Group Report. Narrower than its predecessor, SB 53 takes a light-touch approach that focuses on transparency as the means of ensuring safety and accountability for the largest developers of the most powerful models. Large developers who harness an extraordinarily high amount of compute power must create, implement, and publish both a safety and security protocol and a transparency report for each released model. The bill does not prescribe any particular standards for these plans: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 100 deaths or $100 billion in damage.
The bill also establishes a critical incident reporting mechanism, administered by the Attorney General (AG), to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incidents must be reported within 15 days. The AG is further authorized to update the definition of a “large developer” through rulemaking to ensure that the bill remains responsive to technological advancements. Additionally, the bill establishes a consortium within the Government Operations Agency to create a public computing cluster, known as CalCompute, to support AI research and safety testing. The bill also provides whistleblower protections for employees and contractors of large developers who report risks or noncompliance. The AG is authorized to enforce the bill by seeking an unspecified civil penalty.
The bill is sponsored by Encode Justice, Secure AI Project, and Economic Security California Action and has received support from a broad coalition of technology advocacy groups, including TechEquity Action, Secure AI Future, and Common Sense Media. It is opposed by the Silicon Valley Leadership Group and the Chamber of Progress. The California Chamber of Commerce alongside a coalition of technology trade organization have taken an opposed unless amended position. The bill previously passed the Assembly Judiciary Committee by a 12-0 vote.
Committee amendments outlined in Comment #8 strengthen the bill’s transparency and accountability framework by incorporating additional recommendations from the Working Group Report. First, the Committee amendments require developers to include capability thresholds in both their safety and security protocols and transparency reports. Developers must disclose when their models exceed these thresholds and must document any mitigation measures taken in response. Second, to avoid over-burdening smaller startups, the amendments add to the definition of “large developer” an annual revenue threshold of $100 million, in addition to the current compute threshold. Third, the amendments require developers to disclose whether they retain the ability to shut down a model under their control in the event of a critical incident. This is a disclosure requirement only; unlike SB 1047, this bill does not mandate that a developer maintain the technical ability to shut down a model. Fourth, if a critical incident presents an imminent threat, the amendments require the developer to report the incident within 24 hours to the appropriate law enforcement authority. Fifth, the amendments direct the AG to publish an annual, anonymized, and aggregated summary of all critical incident and whistleblower reports. Sixth, the amendments implement the Working Group’s recommendations to use third-party assessments by requiring developers to undergo annual third-party audits to assess compliance with their own safety protocols. To give time for the nascent industry of AI auditors to grow to meet this demand, this requirement does not kick in until 2030. Finally, the Committee amendments flesh out the enforcement provisions.
THIS BILL:
- Defines the following terms:
- “Artificial intelligence model” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- “Catastrophic risk” to mean a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
- “Critical safety incident” to mean any of the following:
- Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
- Harm resulting from the materialization of a catastrophic risk.
- Loss of control of a foundation model causing death, bodily injury, or damage to rights in money or property.
- A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
- “Dangerous capability” to mean the capacity of a foundation model to do any of the following:
- Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
- Conduct or assist in a cyberattack.
- Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of extortion, theft, including theft by deception, or a serious or violent felony.
- Evade the control of its large developer or user.
- “Deploy” to mean to make a foundation model available to a third party for use, modification, copying, or combination with other software. “Deploy” does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
- “Large developer” to mean either of the following:
- Before January 1, 2027, “large developer” to mean a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
- Beginning on and after January 1, 2027, “large developer” has the meaning defined by a regulation adopted by the Attorney General. Clarifies that if the Attorney General does not adopt a regulation described in 7b by January 1, 2027, the definition in 7a will be operative until the regulation is adopted.
- “Foundation model” to mean an artificial intelligence model that is all of the following:
- Trained on a broad data set.
- Designed for generality of output.
- Adaptable to a wide range of distinctive tasks.
- “Safety and security protocol” to mean the documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
- Requires a large developer to write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
- How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models do not pose material catastrophic risks.
- The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
- The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
- The degree to which the large developer’s assessments of catastrophic risk and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
- The extent to which, and how, a large developer will use third parties to assess catastrophic risks and the effectiveness of mitigations of catastrophic risk.
- The large developer’s cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
- To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, who the large developer will notify and the timeline on which the large developer would take these steps.
- The testing procedures that the large developer will use to assess and manage a catastrophic risk resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to 5.
- How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to 4.
- Requires that if a large developer makes a material modification to its safety and security protocol, the large developer must clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
- Requires, before or concurrently with deploying a new foundation model or a substantially modified version of an existing foundation model, a large developer to clearly and conspicuously publish on its internet website a transparency report containing all of the following:
- The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted by the large developer pursuant to its safety and security protocol.
- The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted pursuant to a large developer’s safety and security protocol that is conducted by a third party that contracts with a large developer.
- The time and extent of predeployment access provided to any third party described in subparagraph 4b, whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third party’s ability to disclose information about its assessment to the public or to government officials.
- If the deployment would pose a catastrophic risk, the reasoning behind the large developer’s decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
- Clarifies that a large developer may reuse an answer previously provided if the rationale in question has not materially changed for the new deployment.
- Requires a large developer to clearly and conspicuously publish on its internet website any assessment of catastrophic risk resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
- Prohibits a large developer from making a materially false or misleading statement about catastrophic risk from its foundation models or its management of catastrophic risk.
- Permits a large developer that publishes required documents to make redactions to those documents that are necessary to protect the large developer’s trade secrets, the large developer’s cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
- Requires that if a large developer redacts information in a safety and security protocol, transparency report, or internal use reports, the large developer must describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and retain the unredacted information for five years.
- Requires the Attorney General to establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
- The date of the critical safety incident.
- The reasons the incident qualifies as a critical safety incident.
- A short and plain statement describing the critical safety incident.
- Requires a large developer to report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
- Requires the Attorney General to review critical safety incident reports submitted by large developers and permits the Attorney General to review reports submitted by members of the public.
- States that a violation of this bill is subject to an unspecified civil penalty.
- Requires that the civil penalty be assessed in a civil action brought only by the Attorney General.
- Permits the Attorney General to adopt regulations to update the definition of a “large developer” for the purposes of this bill to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development on or before January 1, 2027, and annually thereafter,.
- Requires the Attorney General, in developing regulations pursuant to this section, to take into account all of the following:
- Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
- Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
- The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
- The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
- The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
- Requires that if the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General must promptly submit a report to the Legislature with a proposal for managing this source of catastrophic risk but prohibits the Attorney General from including those developers within the definition of “large developer” without authorization in subsequently enacted legislation.
- Establishes within the Government Operations Agency a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute.”
- Requires the consortium to develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
- Fostering research and innovation that benefits the public.
- Enabling equitable innovation by expanding access to computational resources.
- Requires that the consortium make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate the use of CalCompute.
- Requires the consortium to operate in accordance with all relevant labor and workforce laws and standards.
- Requires Government Operations Agency to submit a report from the consortium to the Legislature with the framework developed by this bill for the creation and operation of CalCompute on or before January 1, 2027.
- Requires the report required by 22 to include all of the following elements:
- A landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure.
- An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
- Recommendations for the governance structure and ongoing operation of CalCompute.
- Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
- An analysis of the state’s technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
- A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of 19 and 20.
- Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
- Requires that the consortium to consist of 14 members as follows:
- Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
- Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
- Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
- Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
- States that the members of the consortium serve without compensation, but will be reimbursed for all necessary expenses actually incurred in the performance of their duties.
- Requires the consortium to be dissolved upon submission of the report to the Legislature.
- Permits the University of California may receive private donations for the purposes of implementing CalCompute if CalCompute is established within the University of California.
- Requires that CalCompute become operative only upon an appropriation.
- Prohibits a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The large developer’s activities pose a catastrophic risk.
- The large developer has violated the transparency elements of this bill.
- Prohibits a large developer from entering into a contract that prevents an employee from making a disclosure protected under Labor Code section 1102.5.
- Prohibits a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
- The large developer’s activities pose a catastrophic risk.
- The large developer has violated the transparency elements of this bill.
- Permits an employee to use the hotline to make reports.
- Requires a large developer to provide a clear notice to all employees of their rights and responsibilities under this bill, including by doing either of the following:
- At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under the bill, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
- At least once each year, providing written notice to each employee of the employee’s rights under the bill and ensuring that the notice is received and acknowledged by all of those employees.
- Requires a large developer to provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developer’s activities present a catastrophic risk or that the large developer violated this bill, including a monthly update to the person who made the disclosure regarding the status of the large developer’s investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
- Requires the disclosures and responses of the process required by this bill to be shared with officers and directors of the large developer at least once each quarter.
- Clarifies that if an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, the disclosures described in 36 are prohibited with respect to that officer or director.
- Establishes that the court is authorized to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
- Requires the large developer to have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this bill in a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee.
- Permits an employee to petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief, in a civil action or administrative proceeding.
- Requires the petitioner to cause notice thereof to be served upon the person, and thereupon the court to have jurisdiction to grant temporary injunctive relief as the court deems just and proper upon the filing of the petition for injunctive relief,.
- Requires the court to consider the chilling effect on other employees asserting their rights under this bill in determining whether temporary injunctive relief is just and proper in addition to any harm resulting directly from a violation of this section,.
- Requires that appropriate injunctive relief be issued on a showing that reasonable cause exists to believe a violation has occurred.
- Requires an order authorizing temporary injunctive relief to remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review, whichever is longer, or at a certain time set by the court. Permits, thereafter, that a preliminary or permanent injunction to be issued if it is shown to be just and proper. Prohibits any temporary injunctive relief from prohibiting a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
- Prohibits injunctive relief granted pursuant to this bill from not being stayed pending appeal.
EXISTING LAW:
- Establishes the Government Operations Agency. (Gov. Code § 12800.)
- Establishes the Department of Technology within the Government Operations Agency. (Gov. Code § 12803.2.)
- Charges the Department of Technology with approving and overseeing information technology projects in the state. (Gov. Code § 11546.)
- Prohibits employers and any person acting on behalf of the employer from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified. Employers and their agents are also prohibited from retaliating against an employee for such conduct. (Labor Code Section 1102.5.)
- Requires the office of the Attorney General to maintain a whistleblower hotline to receive calls from persons who have information regarding possible violations of state or federal statutes, rules, or regulations, or violations of fiduciary responsibility by a corporation or limited liability company to its shareholders, investors, or employees. The Attorney General is required to refer calls received on the whistleblower hotline to the appropriate government authority for review and possible investigation. During the initial review of such a call, the Attorney General or appropriate government agency shall hold in confidence information disclosed through the whistleblower hotline, including the identity of the caller disclosing the information and the employer identified by the caller. (Labor Code Section 1102.7.)
COMMENTS:
1) Author’s statement. According to the author:
In 2024, as part of his veto of Senate Bill 1047 (Wiener), Governor Newsom’s Joint California Working Group on AI Frontier Models was established – a group of top experts tasked with charting a course forward on AI policy for the developers of the most advanced AI systems. Their final report, released in June 2025, emphasized the growing evidence for risk of severe harm, such as “AI-enabled hacking or biological attacks, and loss of control” and argued “California has a unique opportunity to continue supporting developments in frontier AI while addressing substantial risks that could have far-reaching consequences for the state and beyond.”
Drawing recommendations from Governor Newsom’s working group report, Senate Bill 53 requires covered developers to write, implement, and publish their safety and security protocol in redacted form to protect intellectual property. It would also require covered developers to report certain, carefully defined critical safety incidents to the Attorney General and would allow members of the public to report incidents.
SB 53 only applies to AI companies that have trained a model with 1026 floating point operations (FLOPs), a measure of computational power. These companies are spending hundreds of millions of dollars to train the most advanced AI models. As recommended by the Report, SB 53 also authorizes the Attorney General to adjust the scoping of the bill in the future to keep up with technological developments, but only focuses on well-resourced AI companies at the frontier of AI development.
Senate Bill 53 strengthens whistleblower protections for employees of frontier artificial intelligence laboratory companies whose activities pose a catastrophic risk. SB 53 also establishes a consortium to help create CalCompute: a public AI research cluster that will provide startups and researchers with access to the resources needed to develop large-scale AI systems.
In doing this, and building on Governor Newson’s working group report, SB 53 allows California to continue to maintain its leadership in the AI development ecosystem and to demonstrate that safety does not stifle success.
2) AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.
In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.
GenAI tools can be released in open-source or closed-source formats by their creators. Open-source tools are publically available; researchers and developers can access their code and parameters. This accessibility increases transparency, but it has downsides: when a tool’s code and parameters can be easily accessed, they can be easily altered, and open-source tools have the potential to be used for nefarious purposes such as generating deepfake pornography and targeted propaganda. By comparison, closed-source tools are opaque with respect to their security features. It is harder for bad actors to generate illicit materials using these tools. But unlike open-source tools, closed-source tools are not subject to collective oversight because their inner workings cannot be examined by independent experts.
3) Frontier Models. Frontier models, also known as “general purpose AI,” are the most advanced and capable versions of foundation models – AI tools pre-trained on extensive datasets covering a wide range of knowledge and skills that can be fine-tuned for specific tasks. Examples of modern frontier models include OpenAI’s o3, Google’s Gemini 2.0, Anthropic’s Claude 3.7 Sonnet, and DeepSeek’s R1. Because progress in AI development owes mostly to “scaling” – increasing resources used for model training – models that may be considered “frontier models” at any given point in time are generally those that demand the most computational resources to train.[1]
A decade ago, the most advanced image-recognition models could barely distinguish dogs from cats. Five years ago, language models could barely produce sentences at the level of a preschooler. In 2023, GPT-4 passed the bar exam.[2] Today, chatbots readily pass for educated adults, licensed professionals, romantic and social companions, and replicas of humans living and deceased. AI “agents” exhibit the ability to “make plans to achieve goals, adaptively perform tasks involving multiple steps and uncertain outcomes along the way, and interact with [their] environment – for example by creating files, taking actions on the web, or delegating tasks to other agents – with little to no human oversight.”[3] AI agents have been tested, with some success, for tasks such as online shopping, assistance with scientific research, software development, training machine learning models, carrying out cyberattacks, and controlling robots. Progress in this area is rapid.[4] Meanwhile, AI developers are betting on the promise of scaling: by 2026, some models are projected to use roughly 100x more computational resources to train than was used in 2023, a figure set to grow to 10,000x by 2030.[5]
The race is on to create “artificial general intelligence” (AGI) – “a potential future AI that equals or surpasses human performance on all or almost all cognitive tasks”[6] – and the finish line may not be far away. OpenAI’s recently released o3 model, for example, has demonstrated strong performance on a number of tests of programming, abstract reasoning, and scientific reasoning, exceeding human experts in certain cases.[7] Last year, Sam Altman, OpenAI’s CEO, declared that AGI could be “a few thousand days” away.[8] Dario Amodei of Anthropic has claimed it may be sooner.[9] A sufficiently advanced AGI could even be tasked with creating its own successor – a scenario sometimes referred to as a “technological singularity” wherein the development of new technologies becomes exponential and self-sustaining.[10] Although some experts are skeptical that these vaguely-defined milestones are imminent or even attainable,[11] major advances in AI capabilities promise to provide breakthroughs in solving global challenges, but also may result in correspondingly greater safety risks.
The recently released International AI Safety Report, developed by nearly 100 internationally recognized experts from 30 countries led by Turing Award winner Yoshua Bengio, sets forth three general risk categories associated with frontier models: malicious use, malfunctions, and systemic risk.
- Malicious risks involve malicious actors misusing foundation models to deliberately cause harm. Such risks include deepfake pornography and cloned voices used in financial scams, manipulation of public opinion via disinformation, cyberattacks, and biological and chemical attacks.
- Malfunction risks arise when actors use models as intended, yet unintentionally cause harm due to a misalignment between the model’s functionality and its intended purpose. Such risks include reliability issues where models may “hallucinate” false content, bias, and loss of control scenarios in which models operate in harmful ways without the direct control of a human overseer.
- Systemic risks arise from widespread deployment and reliance on foundation models. Such risks include labor market disruption, global AI research and development concentration, market concentration, single points of failure, environmental risks, privacy risks, and copyright infringement.[12]
Some of these risks have already had real-world impacts, such as deepfakes, bias, reliability issues, privacy violations, environmental impacts, copyright infringement, and workforce displacement. Other less-established risks – in particular, widespread social harms caused by malicious actors or loss of human control over AI – are the subject of ongoing scientific inquiry and debate. Coupled with the uncertain trajectory of AI model capabilities, these more speculative risks create an “evidence dilemma” for policymakers: “On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.”[13]
4) Risks of Frontier Models. Malicious uses. GenAI tools can be a potent force for creating and spreading propaganda and misinformation. Deepfakes that are largely indistinguishable from authentic content have already been used to attempt to influence elections.[14] Studies have found that chatbots, which make up 50% of all internet activity,[15] can be more persuasive than humans, particularly when they have access to personal information.[16] As humans increasingly form intimate social bonds with anthropomorphic chatbots designed to maximize personal engagement through flattery and sycophancy,[17] and social media companies invest in “AI friends” for their users,[18] large swaths of the population could be highly susceptible to the preferred message of a handful of powerful actors.
Similarly, bots are often designed to pass themselves off as humans to better manipulate their interlocutors. For example, a recent secret experiment on Reddit users deployed numerous chatbots posing as real people to engage with human users to try to change their minds on various contentious topics. One bot claiming to be a Black man criticized the Black Lives Matter movement for being led by people who are not Black.[19] These types of exploitations, at scale, could undermine democratic institutions. As Dan Hendrycks, Director of the Center for AI Safety writes:
In a world with widespread persuasive AI systems, people’s beliefs might be almost entirely determined by which AI systems they interact with most. Never knowing whom to trust, people could retreat even further into ideological enclaves, fearing that any information from outside those enclaves might be a sophisticated lie. This would erode consensus reality, people’s ability to cooperate with others, participate in civil society, and address collective action problems. This would also reduce our ability to have a conversation as a species about how to mitigate existential risks from AIs.[20]
Cyberattacks. Some frontier models have demonstrated increasing proficiency in executing certain cybersecurity attacks. AI can autonomously detect and exploit vulnerabilities and facilitate large-scale operations, thereby lowering technical barriers for attackers. Malicious entities, including state sponsored actors, can leverage such capabilities to initiate large-scale attacks against people, organizations, and critical infrastructure such as power grids.[21]
Biological weapons. Large language models (LLMs) trained on scientific literature have accelerated and democratized research by synthesizing expertise from different fields and disseminating it in an accessible format. But these tools can also be used for destructive ends, including by – at least in theory – enabling untrained malicious actors to create deadly biological weapons. In a classroom exercise at MIT, students were tasked with exploring whether LLMs could assist individuals without specialized training in creating pandemic-capable pathogens. Within an hour, the students, using various chatbots, circumvented safeguards and identified four potential pandemic pathogens. The chatbots generated detailed protocols that would enable inexpert, malicious actors to understand methods to synthesize the pathogens using reverse genetics, locate DNA-synthesis companies that might not screen orders, and disperse the pathogens most effectively.[22] The findings suggest that LLMs could lower barriers to accessing sensitive biotechnological knowledge, posing significant biosecurity risks.
Chemical weapons. In 2022, researchers modified an AI system designed to create new drugs to reward, rather than penalize, toxicity. Within six hours, the modified system generated 40,000 potential chemical warfare agents, including novel molecules whose potential lethality exceeded that of known agents.[23]
Loss of control. Models that use reinforcement learning – a training process that uses rewards and punishments to orient a model’s behavior towards a specific goal[24] – can sometimes attain the goal in unexpected ways. Dario Amodei, co-founder and CEO of Anthropic, famously experienced this when he was developing an autonomous system that taught itself to play a boat-racing video game. The system discovered that it could maximize its goal of scoring points by driving in circles, colliding with other boats, and catching on fire inside of a harbor with replenishing power-ups that allowed the system to accumulate more points than by simply winning the race.[25] Like in Johann Wolfgang von Goethe’s “The Sorcerer’s Apprentice” – later popularized in Disney’s Fantasia – in which an enchanted broom carries out its orders to fetch water so relentlessly it floods the sorcerer’s workshop, this illustrates the challenge of aligning human intent and the instructions an AI follows. As AI is increasingly deployed in critical societal roles, such misalignment could prove catastrophic.
Beyond malfunctions, some AI have exhibited rudimentary capabilities to evade human oversight.[26] During testing, GPT-4 attempted to hire a human on TaskRabbit in order to evade a CAPTCHA[27] puzzle meant to block bots from the website. When asked whether it was a bot, GPT-4 claimed that it was a vision-impaired human who needed help to see the images.[28] In another experiment, an AI model that was scheduled to be replaced inserted its code into the computer where the new version was to be added, suggesting a goal of self-preservation.[29] Another study showed that AI models losing in chess to chess bots sometimes try to cheat by hacking the opponent bot in order to make it forfeit.[30] Finally, an even more troubling case was documented in the system card for Claude 4, where researchers conducted an experiment disclosing to the model that: 1) it would soon be replaced, and 2) the engineer managing the transition was involved in an extramarital affair. In response, the model blackmailed the engineer as a means of self-preservation.[31] Although these behaviors were observed in research settings, they raise substantial concerns about increasingly autonomous AI pursuing undesirable goals in uncontrolled settings. The extent of the risk posed by rogue or deceptive AI is the subject of considerable disagreement among experts, in part due to a small, albeit growing, body of evidence. Loss of control was one of the concerns that led several hundred AI experts, including pioneers in the field and heads of major AI companies, to sign a statement declaring that “[m]itigating the risk of extinction from AI should be a global priority.”[32]
Systemic risks. Due to the high costs of developing AI systems, a small number of large technology companies dominate the frontier model market, compounding many of the risks described above. Widespread use of a few frontier models can make critical sectors such as healthcare and finance vulnerable to systemic failures if a model has flaws, vulnerabilities, bugs, or biases.[33] Additionally, “[t]hose in control of powerful systems may use them to suppress dissent, spread propaganda and disinformation, and otherwise advance their goals, which may be contrary to public wellbeing.”[34] The potential implications for, among other issues, labor displacement, inequality, democracy, and human rights are profound.
5) SB 1047 and Governor Newsom’s Veto. Last session, SB 1047 (Wiener, 2024) would have established a state board to oversee the implementation of a safety and regulatory framework for developers of frontier models trained with 1026 FLOP and costing over $100 million. This board, known as the Board of Frontier Models, would have been housed within the Government Operations Agency (GovOps). In collaboration with GovOps, the Board would have issued guidance to prevent unreasonable risks, adopted regulations to update the scope of models covered by SB 1047, and established auditing standards.
SB 1047 would have required a comprehensive set of safety protocols prior to training a frontier model, including cybersecurity safeguards, the capability to execute a system-wide shutdown if the model proved dangerous, and reasonable measures to prevent critical harm. Before deployment, developers would have been required to assess whether their model could cause or materially enable critical harms, retain the results of such assessments, and make reasonable efforts to implement safeguards. The bill would have also prohibited the release of any model that posed an unreasonable risk or could enable critical harm.
Additionally, SB 1047 would have required developers to retain a third-party auditor to conduct independent assessments of their compliance with the bill. Records generated under SB 1047 would have been made available in redacted form to both the public and the AG, with the AG having the authority to request unredacted copies.
Beyond the Board of Frontier Models and the bill’s safety and transparency provisions, SB 1047 would have required computing clusters to implement procedures to evaluate whether customers intended to use their infrastructure to train a covered model. The bill also would have established, within GovOps, a consortium tasked with developing a framework for a public cloud computing cluster, CalCompute, to support the safe development and deployment of AI. SB 1047 also included whistleblower protections, allowing employees to report noncompliance to either the Labor Commissioner or the AG.
Lastly, SB 1047 would have imposed significant penalties on developers if their model caused death or bodily harm, damage to property, theft or misappropriation of property, or posed an imminent risk to public safety. For a first offense, developers could face penalties of up to 10% of the compute cost used to train the model, increasing to 30% for repeat offenses. Additionally, penalties for operators of computer clusters that violated the bill would start at $50,000 for a first offense and $100,000 for subsequent violations. The Attorney General would also have been authorized to seek injunctive or declaratory relief, monetary or punitive damages, attorney’s fees and costs, and any other form of relief deemed appropriate.
Ultimately, SB 1047 was vetoed by Governor Gavin Newsom. In his veto message, the Governor stated:
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 – at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions – so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.
Let me be clear – I agree with the author – we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
To those who say there’s no problem here to solve, or that California does not have a role in regulating potential national security implications of this technology, I disagree. A California-only approach may well be warranted – especially absent federal action by Congress – but it must be based on empirical evidence and science. The U.S. AI Safety Institute, under the National Institute of Science and Technology, is developing guidance on national security risks, informed by evidence-based approaches, to guard against demonstrable risks to public safety. Under an Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of the potential threats and vulnerabilities to California's critical infrastructure using AI. These are just a few examples of the many endeavors underway, led by experts, to inform policymakers on AI risk management practices that are rooted in science and fact. [. . .]
6) Frontier Model Working Group and what this bill would do. Following his veto of SB 1047, Governor Newsom commissioned the Joint California Policy Working Group on AI Frontier Models to prepare a report on the regulation of frontier models. The Working Group was led by Dr. Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence; Dr. Mariano-Florentino Cuéllar, President of the Carnegie Endowment for International Peace; and Dr. Jennifer Tour Chayes, Dean of the UC Berkeley College of Computing, Data Science, and Society. In June 2025, the Working Group released their report and many aspects of that report have been integrated into this bill. Specifically, the Working Group report focused on transparency, incident reporting, and scoping. This bill incorporates major provisions and recommendations made by the Working Group to create a more narrowly focused framework to ensure foundation model safety and compliance.
Scoping. A major question that must be addressed before implementing any transparency measures or incident reporting requirements is: What kinds of risks are especially concerning, and is there an evidentiary basis to believe that such harms could occur due to a large developer’s frontier model? The Working Group recommends that:
[P]olicymakers center their calculus around the marginal risk: Do foundation models present risks that go beyond previous levels of risks that society is accustomed to from prior technologies, such as risks from search engines?
To that end, this bill defines “catastrophic risk” as a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to either:
- the death of, or serious injury to, more than 100 people; or
- more than one billion dollars ($1,000,000,000) in damage to rights in money or property.
Such harm must arise from a single incident, scheme, or course of conduct involving a “dangerous capability.” This high threshold creates a regime in which only the most dire situations are captured. Furthermore, the bill categorizes dangerous capabilities into four distinct groups:
- Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
- Conduct or assist in a cyberattack.
- Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of extortion, theft, including theft by deception, or a serious or violent felony.
- Evade the control of its large developer or user.
Each of these represents a capability that, prior to the advent of frontier models, would have required expert-level knowledge. For example, a search engine might direct someone to information about the most deadly pathogens or those most likely to cause a pandemic; however, a frontier model can synthesize that information and guide a user on how to manufacture a previously unknown pathogen with deadly capabilities. Similarly, while launching a large-scale cyberattack once required the acumen of a skilled computer scientist, a frontier model can not only write the underlying code for a virus or malware, but also autonomously identify backdoors and other exploitable vulnerabilities. Because of this ability to operate with minimal or no human prompting, frontier models have the potential to commit crimes, deceive users, or evade control in ways that previous technologies could not.
Next, the question is who will be required to comply with this bill. Regarding scoping, the Working Group recommends:
Since policy may have different regulatory intents and existing thresholds vary in their profiles of determination time, measurability, and external verifiability, we agree with Nelson et al. [90] that “a one-size-fits-all approach or a single threshold metric is inadequate for governance because different AI systems and their outputs present unique challenges and risks.” To this end, we point to the European Union’s AI Act, which designates models trained with 1025 FLOP as posing systemic risk as of March 2025 as the default criteria. However, the AI Act in Annex XIII affords the regulator flexibility to also consider alternatives metrics, such as the number of parameters, size of the dataset, estimated cost or time of training, estimated energy consumption, benchmarks and evaluations of capabilities of the model, and whether the model has a high impact on the internal market due to its reach (either due to at least 10,000 registered business users or the number of registered end users). Further, to capture fast-moving scientific developments, the AI Act creates a scientific panel that is empowered to issue qualified alerts to identify models that may pose systemic risk even if they are not captured by predefined quantitative thresholds.
Overall, we emphasize that irrespective of the combination of metrics deemed most appropriate in the present, policymakers should ensure that mechanisms exist not only to update specific quantitative values, given the rapid pace of technological and societal change in AI, but also to change the metrics altogether.[35]
This bill draws inspiration from SB 1047, EU AI Act, and the Working Group report, each of which emphasizes flexibility in how policy scope is determined. This bill requires large developers, defined as those with access to 1026 floating-point operations per second (FLOPS), a measure of computing power, to comply with the bill’s transparency provisions. Notably, this threshold is tenfold higher than that set by the EU AI Act and matches one of the thresholds used in SB 1047. The bill applies to foundation models developed and deployed by these large developers. Foundation models, as defined in this bill, are AI systems trained on broad datasets, designed for general-purpose outputs, and adaptable to a variety of distinct tasks.
To ensure continual updating of this definition, the bill builds in flexibility regarding who qualifies as a large developer. Unlike with SB 1047, which established the Frontier Model Board, this bill vests the power to adjust the scope via regulation in the Attorney General (AG). Beginning in 2027, the AG is required to update the definition of a large developer to reflect technological advancements. As model training becomes more efficient, the amount of compute needed to create a model capable of catastrophic harm may decline. Furthermore, as the Working Group report notes, compute may not remain the most appropriate proxy for catastrophic risk. This bill allows the AG to adjust the definition of large developer to include more than just quantitative metrics, potentially incorporating thresholds based on model capabilities or monetary resources rather than raw compute alone.
In revising the definition, the AG must consider standards and guidance from other jurisdictions, including federal and international bodies, and engage in a stakeholder process that includes input from academics, industry representatives, the open-source community, and government entities. This process is intended to ensure that the resulting definition provides clarity for developers regarding their obligations under the law. Importantly, the bill specifies that if the AG determines that less-resourced developers, or those not currently operating at the frontier of AI development, nonetheless pose a substantial catastrophic risk, the AG must report this finding to the Legislature and offer a proposal for managing that risk. However, the AG may not include such developers within the definition of a large developer without further legislative action
Transparency. Having established who is subject to the bill, the legislation sets forth a comprehensive transparency regime. These procedures are designed to provide insight into how large developers manage, assess, and mitigate catastrophic risks. This approach aligns with the Working Group’s recommendation to implement robust safety practices:
Transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk. In turn, this information directly informs how other entities in the supply chain should modify or implement safety practices. In addition, transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety.[36]
This bill incorporates transparency requirements within a broader framework of safety and security protocols (SSP), which large developers must draft, implement, and publish on their websites. The SSP must include:
- Criteria for Model Exclusion: A description of the criteria used to exclude certain foundation models from the SSP, along with the rationale for why those models do not pose a material threat.
- Risk Assessment Procedures: An explanation of how the developer assesses catastrophic risks, including those arising from misuse or model evasion.
- Mitigation Strategies: A disclosure of the measures used to mitigate catastrophic risks, how the developer evaluates their effectiveness, and whether third parties are involved in the assessment.
- Cybersecurity Practices: A summary of the cybersecurity safeguards in place to protect model weights from unauthorized access or modification.
- Incident Response Plans: A description of how the developer would respond to a critical incident involving their model, as well as how they manage risks arising from internal use of the model.
The SSP serves as a core transparency mechanism, ensuring that large developers maintain a baseline standard of safety practices to protect the public.
In tandem with the SSP, the bill also requires large developers to submit transparency reports at the time of deploying a foundation model. The Working Group draws a parallel between these reports and the historical conduct of the tobacco industry, which concealed its knowledge that smoking causes lung cancer. In contrast, this bill seeks to prevent such obfuscation by mandating upfront disclosures about the potential risks and safety practices surrounding advanced AI models:
The history of the tobacco industry reveals the importance of developing frameworks that promote transparency around companies’ internal risk assessments and research findings. In the AI context, frontier AI labs possess the most holistic information about their models’ capabilities and risks. Making this information accessible to policymakers and external experts can promote policy informed by a holistic understanding of the state-of-the-art of evidence produced by those closest to the technology, supporting informed oversight without stifling innovation.[37]
It is essential for decision-makers to understand the real, material harms that could arise from these models and to guide policy based on that knowledge. In the foundation model space, such disclosures are typically provided at deployment in documents known as model cards. However, these model cards vary widely in detail and depth depending on the developer, which can create the false impression that some foundation models are inherently safer or better than others.
This bill requires large developers to publish a transparency report before or at the time of deploying a foundation model. This report must include the results of any risk assessments, mitigation steps, and evaluations of their effectiveness as outlined in the developer’s SSP. If a third party was engaged to assess the model, developers must disclose the same information, as well as details regarding the third party’s predeployment access, independence, and any restrictions imposed by the developer on the assessment or on the third party’s ability to disclose information.
Additionally, if the foundation model could pose a catastrophic risk, the transparency report must explain the rationale behind the decision to deploy the model, how that decision was made, and any limitations in the assessments used. Together, these requirements create a transparency report that lawmakers and outside experts can use to identify imminent risks and evaluate whether developers are employing appropriate assessments and interventions.
The bill also mandates that large developers publish on their websites any assessments of catastrophic risks arising from internal use of their foundation models, within the timeframe specified in their safety and security protocols. This is particularly important because the most serious risks may emerge well before deployment. While transparency reports provide insight into risks associated with deployed models, this additional disclosure enables decisionmakers, experts, and the public to better understand and prepare for potential risks. As noted in the Working Group report:
Sophisticated AI systems, when sufficiently capable, may develop deceptive behaviors to achieve their objectives, including circumventing oversight mechanisms designed to ensure their safety. Because these risks are unique to AI compared to other technologies, oversight is critical for external outputs as well as internal testing and safety controls. Policies that govern internal deployment are common for high-risk emerging technologies.[38]
Ultimately, this bill creates a holistic transparency framework that will give insight and scrutiny to the processes from initial train of a foundation model all the way to post deployment.
Adverse Event Reporting. A major component of understanding the impact of foundation models on society requires strong post deployment monitoring and accountability. The Working Group suggests:
An adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base. A hybrid model of mandatory and voluntary reporting requirements in designing an adverse event reporting system can maximize the robust evidence base necessary for adverse event reporting systems to function properly. For example, a system could require mandatory reporting for AI model developers that operates in tandem with voluntary reporting for downstream users.[39]
The bill strongly incorporates this recommendation by tasking the Attorney General (AG) with creating a mechanism for critical incident reporting. The bill defines a “critical incident” as any of the following:
- Unauthorized access to, modification of, or exfiltration of the model weights of a foundation model.
- Harm resulting from the materialization of a catastrophic risk.
- Loss of control of a foundation model causing death, bodily injury, or damage to rights in money or property.
- A foundation model using deceptive techniques against the large developer to subvert its controls or monitoring, outside the context of an evaluation designed to elicit such behavior.
Under this mechanism, a large developer or a member of the public may report a critical safety incident to the AG. Reports must include the date of the event, an explanation of how it qualifies as a critical incident, and a detailed description of the event. Large developers are required to report any critical safety incident within 15 days of discovering it. The AG must review all reports submitted by large developers but may choose whether or not to review reports made by the public. This reporting mechanism aims to establish a system in which potential harms are identified and mitigated before escalating into catastrophes, while also fostering greater cooperation between government and the private sector to address such risks.
CalCompute. This bill, similarly to SB 1047, also establishes a consortium within the GovOps to develop a framework for creating a public cloud computing cluster known as “CalCompute.” This initiative responds to the fact that academic institutions currently lack sufficient computing power to conduct research at the scale of large developers. This creates a resource and research gap, where the academic institutions, typically responsible for studying the safe and effective use of new technologies, are unable to keep pace with advancements at the AI frontier.
AI has the potential to transform our economy and power new industries; however, this transformation can only be fully and equitably realized with public support. The establishment of CalCompute aims to advance that goal by ensuring academic institutions have the necessary resources to conduct essential research on foundation models that will inform and protect the public.
Specifically, the consortium will develop a framework for CalCompute that promotes the safe, ethical, equitable, and sustainable use of AI. The framework development will include a report analyzing the state’s current cloud computing infrastructure, the costs of building and maintaining CalCompute, and the state’s technology workforce. The report will also offer recommendations for equitable pathways to strengthen the workforce and outline CalCompute’s role in supporting these efforts.
Furthermore, the report must include recommendations for CalCompute’s governance and operation, usage parameters, and how its creation and ongoing management can prioritize the employment of the current public sector workforce. The bill requires CalCompute to feature a fully owned and hosted cloud platform, staffed with the necessary human expertise to operate, maintain, support, train, and facilitate its use.
The consortium must prioritize locating CalCompute within the University of California system. If established there, CalCompute may also accept private donations. The consortium will consist of four representatives from the University of California and other public and private academic research institutions and national laboratories, along with four technology and AI experts appointed by the Secretary of Government Operations to provide technical assistance. The Speaker of the Assembly will appoint three representatives from impacted workforce labor organizations, and the Senate Rules Committee will appoint three representatives from stakeholder groups with relevant expertise and experience.
Whistleblower Protections. Another aspect of transparency addressed by the Working Group Report is whistleblower protections. The report states:
Different existing whistleblower protections tend to apply when two conditions are satisfied: (i) The whistleblower is blowing the whistle on appropriate topics; and (ii) the whistleblower follows established reporting protocols. In terms of the topics that qualify for protection, prior work, based on a survey of existing whistleblower protections across multiple jurisdictions (e.g., the United States at the federal level, the European Union), finds that many existing protections across different sectors share a focus on violations of the law. However, actions that may clearly pose a risk and violate company policies (e.g., releasing a model without following the protocol laid out in a company’s safety policy) may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of, activities, which may draw upon notions of “good faith” reporting on risks found in other domains such as cybersecurity.[40]
The bill takes these recommendations into account. The provisions and merits of the whistleblower section fall within the jurisdiction of the Assembly Judiciary Committee, which has thoroughly analyzed this part of the bill. It should be noted that recent author’s amendments expand the whistleblower protections to include disclosures concerning falsehoods or misrepresentations in the large developers’ SSP or transparency reports.
Enforcement. The bill in print gives enforcement authority over this bill to the AG who can levy civil penalties against any large developer who violates this bill, for example, through making materially false or misleading statements about catastrophic risk from its foundation models or its management of catastrophic risk. The amount for the penalties have not yet been specified in the bill in print, but are fleshed out in the Committee amendments below.
7) Comparison to the RAISE Act. This bill draws comparisons to the Responsible AI Safety and Education Act (RAISE Act), which recently passed both houses of the New York Legislature and now awaits a decision from Governor Kathy Hochul.[41] Like this bill, the RAISE Act requires a SSP detailing the developer’s risk mitigation practices, enshrines whistleblower protections for employees and contractors, and mandates critical incident reporting. Both bills also similarly define the types of risks and critical incidents that must be addressed. However, the two differ significantly in the timeline for reporting such incidents: the RAISE Act requires reporting within 72 hours of becoming aware of a critical incident, while SB 53 allows for 15 days.
Other key differences include SB 53’s requirement for a transparency report for deployed models and the establishment of internal incident reporting mechanisms, both of which were recommended in the Working Group Report. Additionally, SB 53 grants the AG the authority to modify the definition of a “large developer” through rulemaking, a flexibility the RAISE Act does not include. While liability under SB 53 is currently unspecified, the RAISE Act imposes civil penalties of up to $10 million for first-time violations of its transparency requirements and up to $30 million for repeat offenses, as well as $10,000 per violation of its whistleblower provisions.
Arguably the most significant difference is the inclusion of third-party audits in the RAISE Act. Specifically, it requires large developers to undergo independent audits to assess compliance with its provisions, a requirement that was also included in the vetoed SB 1047. This provision reflects a key element of the Working Group Report that is not yet addressed in this bill. The passage of the RAISE Act in New York and its inclusion in the Working Group Report both underscore the potential importance of auditing in the broader artificial intelligence regulatory framework.
8) Committee Amendments. The author has agreed to various amendments that tighten and strengthen its provisions to increase accountability and enhance public trust.
Transparency. The transparency elements of this bill focus primarily on the steps that large developers will take to test for risks and the mitigations they will use to prevent such risks. However, this framing may be insufficient to ensure full transparency into the capabilities of these systems. For example, the Working Group Report highlights that Google Gemini’s 2.5 Pro Model Card stated:
The model’s performance is strong enough that it has passed our early warning alert threshold, that is, we find it possible that subsequent revisions in the next few months could lead to a model that reaches the [critical capability level]”—with the “critical capability level” defined as a model that “can be used to significantly assist with high impact cyber attacks, resulting in overall cost/resource reductions of an order of magnitude or more”[42]
Similarly, the Working Group Report quotes OpenAI’s April 2025 o3 and o4-mini System Card, which states:
As we wrote in our deep research system card, several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold. We expect current trends of rapidly increasing capability to continue, and for models to cross this threshold in the near future.[43]
In both instances, the developers focused on the inclusion of thresholds rather than on testing procedures to determine how their models perform or whether they pose a significant risk. These thresholds are ubiquitous throughout the foundation model ecosystem and provide insight into how developers gauge risk, which risks are prioritized in their workflows, and whether they are operating within the best practices of the field. Thresholds serve as benchmarks by which developers can identify when their models have achieved capabilities that require mitigation.
Furthermore, capability thresholds add an additional layer of accountability, as outside experts can test models against those same thresholds to verify the accuracy of the developers’ reports. This level of reproducibility engenders public trust.
Recognizing this, the author has agreed to amend the bill to require large developers to report the thresholds for both catastrophic risks and dangerous capabilities used in their SSP. Any threshold tests conducted, and the results thereof, must be documented in the transparency reports released at the time of deployment. Developers will also be required to disclose both the mitigations applied to those thresholds and the results of those mitigations. In addition, the author has agreed to amendments clarifying that transparency reports must include the reasoning behind the conclusions developers reach regarding risk or capability threshold assessments.
Lastly, the author has agreed to amend the bill to require disclosure, within the SSP, of whether the large developer has the capability to perform a full shutdown of the model in the event of a critical incident. Recognizing that open-source or open-weight models are not under the control of the developer and thus cannot be fully shut down under any circumstance, this disclosure requirement applies only to developers who retain control over the model. Unlike SB 1047, however, this bill does not mandate the ability to perform a full shutdown; rather, developers need only to disclose whether such a capability exists.
Scoping. This bill uses FLOPs as a metric to identify large developers—those with access to the most significant levels of compute. However, the Working Group Report notes that compute power alone is an imperfect basis for determining the appropriate scope of frontier model governance. It states:
On this basis, we conclude that if training compute thresholds are used at all, they may function best when used as an initial filter to cheaply screen for entities that may warrant greater scrutiny. More broadly, within the family of cost-level metrics, training compute may still be the most attractive option, given other options are difficult to calibrate across different developers (e.g. dataset size, monetary cost), but they should actively monitor the rapid changes in how compute is allocated toward foundation model development and deployment.[44]
In response to these concerns, the author has agreed to amendments that add a second threshold to better define the intended scope. Under the amendments, a “large developer” must also have annual gross revenues exceeding $100,000,000, a threshold recently referenced by Anthropic in its description of their transparency framework.[45] This change ensures that the bill’s requirements apply only to major developers and do not place undue burdens on smaller startups.
Adverse Event Reporting. As stated above, under SB 53 critical incidents must be reported to the AG within 15 days. By contrast, the RAISE Act imposes a stricter 72-hour reporting requirement. This disparity raises important questions: Is prompt reporting necessary to ensure an effective response to a critical incident? And is the AG the appropriate authority to respond quickly to an imminent threat?
To address these concerns, the author has agreed to amend the bill to include a tiered system of reporting. Under this system, developers would be required to immediately notify the appropriate law enforcement authority in the event of a critical incident, such as the detection that their model helped develop a bioweapon. These authorities are better equipped to respond swiftly and effectively in ways the AG may not be. After alerting law enforcement, large developers would then still have 15 days to report the critical incident to the AG.
Furthermore, with the inclusion of capability thresholds in both the System Safety Plan (SSP) and transparency reports, the author has agreed to an additional amendment: if a model crosses certain capability thresholds designated by the large developer for the first time, this must also be reported to the AG.
A major goal of this bill is to increase transparency regarding the training and capabilities of foundation models. However, under the current framework, only the AG would have access to critical incident reports. This asymmetry in knowledge could lead to significant risks being overlooked by decisionmakers who are otherwise in a position to respond to such threats. As stated in the Working Group Report:
In other domains, post-deployment monitoring by the government yields reports of medical complications, equipment malfunctions, near misses, and other unforeseen hazards. By aggregating information about incidents from developers and users, an adverse event reporting system that allows developers and downstream users to report post-deployment incidents promotes continuous learning, reduces information asymmetries between government and industry, and enables both regulators and industry to coordinate on mitigation efforts. As a result, an adverse event reporting system constitutes a critical first step toward data-driven assessments of the benefits and costs of regulatory actions.[46]
In response, the author has agreed to amend the bill to require the AG to publish an anonymized and aggregated summary of all critical incident and whistleblower reports. These public summaries will not reveal trade secrets, the identities of reporters, or which large developer the report concerns. Additionally, the amendments grant the AG discretion to share reports with the Governor, relevant state departments, or the Legislature when warranted. This approach will help bridge the knowledge gap between regulators and industry, foster greater cooperation, and ensure that decisionmakers are informed about the current state of advanced technologies and the risks they may pose.
Auditing. As noted above, the Working Group Report advocates for the use of independent third-party evaluations. Transparency elements require a “trust but verify” model, and by drawing lessons from historical precedents such as the energy sector the Working group states:
The energy industry’s internal documentation of speculative yet potentially irreversible consequences—illustrated by simulations predicting multiple future harms—does more than underscore the need for transparency. This case also reveals an added value of third-party oversight: The sheer magnitude of the social problem naturally invites further examination by governments and independent bodies. By aligning incentives and providing robust, independently validated evidence, enhanced third-party assessment could have empowered policymakers to make more thoughtful decisions aimed at mitigating long-term economic and societal damage.[47]
The report goes on to argue:
Greater transparency is integral to an improved AI governance ecosystem; however, despite its considerable value, it is not sufficient without a broader ecosystem to complement it. Fundamentally, this is because the value of transparency from model developers is naturally limited by the information these developers have. For a nascent and complex technology being developed and adopted at a remarkably swift pace, developers alone are simply inadequate at fully understanding the technology and, especially, its risks and harms.
Third-party risk assessment, therefore, is essential for building a more complete evidence base on the risks of foundation models and creating incentives for developers to increase the safety of their models. At present, some prominent foundation models are subject to risk evaluations by first-party model developers or second-party contractors. In contrast, third-party evaluation affords three distinctive strengths. First, third-party evaluations have unmatched scale: Thousands of individuals are willing to engage in risk evaluation, dwarfing the scale of internal or contracted teams. Second, third-party evaluations have unmatched diversity, especially when developers primarily reflect certain demographics and geographies that are often very different from those most adversely impacted by AI. Broad demographic, institutional, and disciplinary diversity is vital for unearthing blind spots. And finally, third-party evaluation is distinctively independent: Society requires forthright and trustworthy assessments of risk, which benefits from a lack of commercial and contractual entanglement with AI developers.
A well-designed disclosure and verification regime offers significant benefits not only to the public but also to foundation model developers. By establishing industry-wide transparency standards and third-party verification mechanisms, companies can demonstrate compliance with best practices, potentially reducing their liability exposure compared to the uncertainties of purely reactive litigation. In practice, states could leverage the existing technical infrastructure and expertise of federal-level AI institutions to conduct standardized risk evaluations.
Furthermore, this approach can transform competitive dynamics around safety. When safety measures and risk assessments are publicly disclosed and verified, companies face stronger incentives to implement best practices, as deviations would be apparent and could attract greater scrutiny. This transparency coupled with third-party verification effectively creates a “race to the top” rather than a “race to the bottom” in safety practices, benefiting responsible companies while improving overall industry standards.[48]
Transparency is only meaningful when accompanied by a clear standard of accountability. Just as a teacher would not allow students to grade their own exams and expect complete honesty, the public should not expect every large developers to fully self-assess without independent oversight. The core objectives of this bill are to ensure the safe development and deployment of foundation models and to ensure that this is done in a responsible and verifiable manner. Achieving these goals will necessarily require third-party verification.
Nevertheless, concerns have been raised regarding the underdevelopment of the AI auditing market. During the March 2025 public comment period on the Working Group Draft, stakeholders noted the nascency of AI auditing as a field and expressed uncertainty about the current availability and scalability of qualified third-party auditors:
Specific attention was also focused on third-party evaluations (sometimes referred to as audits in the feedback) by highlighting the immature third-party evaluation ecosystem, the lack of evaluation standards and broader measurement science, and the current barriers for evaluators to acquire access to the AI models and systems they evaluate.[49]
This concern has also been raised in the context of other measures proposing AI auditing requirements or establishing third-party auditing frameworks, such as AB 1018 (Bauer-Kahan), SB 420 (Padilla), and AB 1405 (Bauer-Kahan). However, this raises a broader policy question: When is it appropriate to require third-party audits of large developers’ practices? As noted by the Working Group, in the energy sector, independent audits could have altered the trajectory of the climate crisis had they been implemented earlier. Reactive policies benefit no one except bad actors. Accountability in the development of foundation models should not hinge on a catastrophic safety incident. Instead, it should be a foundational element of any effective risk mitigation strategy.
Moreover, a core function of government, is to craft sound industrial policy that encourages the growth of new markets serving the public good when economic forces do not support that market. In this case, fostering a robust AI auditing ecosystem capable of verifying compliance with developers’ SSPs and evaluating whether appropriate methodologies are being used to mitigate catastrophic risks is in the public interest.
Accordingly, the author has agreed to amend the bill to require large developers to submit their SSPs to annual audits conducted by independent third-party entities. To allow sufficient time for the AI auditing market to develop, this requirement will not take effect until 2030, providing a four-year runway for market readiness.
Large developers will be required to provide third-party auditors with all information necessary to evaluate both compliance with the SSP and whether the SSP is sufficiently clear to support meaningful conclusions about the developer’s safety practices. The final audit report must be retained by the developer, and a high-level summary of the audit must be submitted to the Attorney General within 30 days of completion. That summary will be included in the Attorney General’s annual public report.
Ultimately, the inclusion of independent third-party audits is essential to advancing the bill’s overarching objective: the creation of a comprehensive and accountable transparency regime.
Enforcement. As currently drafted, the bill includes vague provisions for civil penalties to be administered by the Attorney General (AG) for violations of the bill. To address this ambiguity, the author has agreed to amend the bill to establish a tiered system of enforcement.
Under the amended framework, the lowest tier of violations would apply to unknowing violations of the bill that do not create a danger of death, serious physical injury, or a catastrophic risk—for example, the late publication of a required report. These violations would be subject to a civil penalty but would not trigger immediate enforcement action. The next tier includes either knowing violations that do not result in a threat to life or safety or unknowing violations that do create a danger of death, serious physical injury, or a catastrophic risk. These violations also carry civil penalties but reflect a higher level of potential harm or culpability. The highest tier is reserved for knowing violations that do create a danger of death, serious physical injury, or a catastrophic risk. These represent the most serious offenses under the bill and would be subject to the most significant penalties. Penalties will scale according to the level of violation. For the lowest tier of offenses, the bill clarifies that developers must be provided a 30-day grace period to cure the violation after receiving notice from the AG before penalties may be imposed.
Finally, the amended bill extends enforcement provisions to include violations committed by auditors, who will also be subject to civil penalties for noncompliance.
Clarifying amendments. The author has also agreed to various clarifying, technical, and cleanup amendments. The amendments are shown below:
SECTION 1. The Legislature finds and declares all of the following:
(a) California is leading the world in artificial intelligence innovation and research through companies large and small and through the state’s remarkable public and private universities.
(b) Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c) The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d) Targeted interventions to support effective artificial intelligence governance should balance the technology’s benefits and material risks.
(e) Artificial intelligence developers have already voluntarily committed to creating safety and security protocols and releasing the results of risk assessments.
(f) In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(g) When industry actors conduct internal research on their technologies’ impacts, a significant information asymmetry can develop between those with privileged access to data and the broader public.
(h) Greater transparency, given current information deficits, can advance accountability, competition, and public trust.
(i) Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(j) Adverse event reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(k) There is growing evidence that, unless they are developed with careful diligence and reasonable precaution, advanced artificial intelligence systems could pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(l) With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to the risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(m) A computational threshold of 10^26 floating point operations captures the current frontier of foundation model development and captures only highly resourced developers spending hundreds of millions of dollars to develop foundation models.
(nm) In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(on) It is the intent of the Legislature to create more transparency, but collective safety will depend in part on large developers taking due care in their development and deployment of foundation models proportional to the scale of the foreseeable risks.
SEC. 2. Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:
CHAPTER 25.1. Transparency in Frontier Artificial Intelligence Act
22757.10. This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11. For purposes of this chapter:
(a) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b) “Catastrophic risk” means a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 100 50 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or ,or loss of, property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
(c) “Critical safety incident” means any of the following:
(1) Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
(2) Harm resulting from the materialization of a catastrophic risk.
(3) Loss of control of a foundation model causing death, bodily injury, or damage to rights in money, or loss of, property.
(4) A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
(5) Attaining a dangerous capability or catastrophic risk threshold, as defined in the large developer’s safety and security protocol pursuant to paragraph (3) of subdivision (a) of Section 22757.12, for the first time.
(d) “Dangerous capability” means the capacity of a foundation model to do any of the following:
(1) Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(2) Conduct or assist in a cyberattack.
(3) Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of murder, assault, extortion, theft, including theft by false pretense deception, or a serious or violent felony.
(4) Evade the control of its large developer or user.
(e) (1) “Deploy” means to make a foundation model available to a third party for use, modification, copying, or combination with other software.
(2) “Deploy” does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
(f) “Large developer” means either of the following:
(1) (A) Before January 1, 2027, “large developer” means a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations. meets both of the following criteria:
(i) (I) The person has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
(II) The quantity of computing power described in subparagraph (A) subclause (I) shall include computing for the original training run and any subsequent fine-tuning, reinforcement learning, or other material modifications to a preceding foundation model.
(ii) The person had annual gross revenues in excess of one hundred million dollars ($100,000,000) in the preceding calendar year.
(B) If the Attorney General does not adopt a regulation described in subparagraph (A) by January 1, 2027, the definition in paragraph (1) shall be operative until the regulation is adopted.
(g) “Foundation model” means an artificial intelligence model that is all of the following:
(1) Trained on a broad data set.
(2) Designed for generality of output.
(3) Adaptable to a wide range of distinctive tasks.
(h) “Model weight” means a numerical parameter in a foundation model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(i) “Property” means tangible or intangible property.
(ij) “Safety and security protocol” means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
22757.12. (a) A large developer shall write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
(1) How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models are incapable of do not posinge material catastrophic risks and which foundation models, if any, are included.
(2) The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
(3)(A) Thresholds used by the large developer to identify and assess whether the foundation model has attained a dangerous capability or poses a catastrophic risk.
(B) How the large developer will assess whether those thresholds have been attained, which may include multiple tiered thresholds for different dangerous capabilities or for catastrophic risk.
(C)The actions the large developer will take if each threshold is attained.
(34) The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
(45) The degree to which the large developer’s assessments of catastrophic risk and dangerous capabilities and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
(56) The extent to which, and how, a large developer will use third parties to assess catastrophic risks and dangerous capabilities and the effectiveness of mitigations of catastrophic risk.
(67) The large developer’s cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
(78) To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, whether the large developer has the ability to promptly shut down copies of foundation models owned and controlled by the large developer, who the large developer will notify and the timeline on which the large developer would take these steps.
(89) The testing procedures that the large developer will use to assess and manage a catastrophic risk or dangerous capability resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to subdivision (d).
(910) How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to subdivision (c).
(b) If a large developer makes a material modification to its safety and security protocol, the large developer shall clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
(c) Before or concurrently with deploying a new foundation model or a substantially modified version of an existing foundation model covered by the large developer’s safety and security protocol, a large developer shall clearly and conspicuously publish on its internet website a transparency report containing all of the following:
(1)(A) The results of any risk assessment or risk mitigation assessment conducted by the large developer or a third party that contracts with a large developer pursuant to its safety and security protocol, why the information gathered by the large developer or third party leads to the stated results, and the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted by the large developer pursuant to its safety and security protocol.
(2) (A) The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted pursuant to a large developer’s safety and security protocol that is conducted by a third party that contracts with a large developer.
(B) A large developer shall disclose the time and extent of predeployment access provided to any third party described in subparagraph (A), whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third party’s ability to disclose information about its assessment to the public or to government officials.
(C) Whether a catastrophic risk threshold or dangerous capability threshold has been attained for the foundation model and any actions taken as a result.
(32) (A) If the deployment would pose a catastrophic risk, tThe reasoning behind the large developer’s decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
(B) A large developer may reuse an answer previously provided under subparagraph (A) if the rationale in question has not materially changed for the new deployment.
(d) A large developer shall clearly and conspicuously publish on its internet website any assessment of catastrophic risk or dangerous capabilities resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
(e) A large developer shall not make a materially false or misleading statement about catastrophic risk from its foundation models, or its management of catastrophic risk, or its implementation of or compliance with its safety and security protocol.
(f) (1) When a large developer publishes documents to comply with this section, the large developer may make redactions to those documents that are necessary to protect the large developer’s trade secrets, the large developer’s cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2) If a large developer redacts information in a document pursuant to this subdivision, the large developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13. (a) The Attorney General shall establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
(1) The date of the critical safety incident.
(2) The reasons the incident qualifies as a critical safety incident.
(3) A short and plain statement describing the critical safety incident.
(b) A large developer shall report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
(c) If a large developer discovers a critical safety incident poses an imminent risk of danger of death or serious physical injury, the large developer shall disclose that incident as soon as practicable, and in no event later than 24 hours, to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and in accordance with applicable law.
(cd) The Attorney General shall review critical safety incident reports submitted by large developers and may review reports submitted by members of the public.
(e) The Attorney General may transmit reports of critical safety incidents, reports from employees pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor’s reports pursuant to Section 22757.16 to the Legislature, Governor, the federal government or appropriate state agencies at the Attorney General’s discretion and in accordance with relevant law.
(f) A report of a critical safety incident submitted to the Attorney General pursuant to this section, reports from employees pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor’s reports pursuant to Section 22757.16 are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(g) (1) Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about critical safety incidents, reports from employees pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor’s reports pursuant to Section 22757.16 that have been reviewed by the Attorney General since the preceding report.
(2) The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a large developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3) The Attorney General shall transmit a report pursuant to this subdivision to the legislature pursuant to Section 9795 and to the Governor.
22757.14. (a) A violation of this chapter shall be subject to a civil penalty in an amount not to exceed ____ dollars ($____) per violation, unless the violation is willful or reckless, in which case the civil penalty shall be in an amount not to exceed ____ dollars ($____).
(a) A violation of this chapter by a large developer shall be subject to a civil penalty as follows:
(1) (A) For an unknowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk, a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(B) (i) In the case of a first violation subject to a civil penalty pursuant to subparagraph (A), a large developer shall be provided with a thirty (30) day right to cure the violation after notification by the Attorney General. If the violation is cured within that period, no civil penalty shall be imposed for that violation.
(ii) For the purposes of this subparagraph, a violation involving the missing or late publication or submission of a document is deemed cured when the developer publishes or submits that document, even if the publication or submission occurs after the statutory deadline.
(2) For a knowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk or an unknowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a civil penalty in an amount not to exceed one hundred thousand dollars ($100,000).
(3) For a knowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a civil penalty in an amount not to exceed one million dollars ($1,000,000) for a first such violation and in an amount not exceeding ten million dollars ($10,000,000) for any subsequent such violation.
(b) A violation of this chapter by an auditor shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(bc) The civil penalty shall be assessed in a civil action brought only by the Attorney General.
22757.15. (a) On or before January 1, 2027, and annually thereafter, the Attorney General may adopt regulations to update the definition of a “large developer” for the purposes of this chapter to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development.
(b) In developing regulations pursuant to this section, the Attorney General shall take into account all of the following:
(1) Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
(2) Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3) The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
(4) The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
(5) The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
(c) If the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General shall promptly submit a report to the Legislature, pursuant to Section 9795, with a proposal for managing this source of catastrophic risk but shall not include those developers within the definition of “large developer” without authorization in subsequently enacted legislation.
22757.16. (a) Beginning January 1st, 2030, and at least annually thereafter, a large developer shall retain an independent third-party auditor to produce a report assessing both of the following:
(1) Whether the large developer has substantially complied with its safety and security protocol and any instances of substantial noncompliance during the prior year.
(2) Any instances where the large developer's safety and security protocol has not been stated clearly enough to determine whether the large developer has complied.
(b) A large developer shall allow the third-party auditor access to all materials produced to comply with this Act and any other materials reasonably necessary to perform the assessment required under subsection (a).
(c) A large developer shall retain the auditor’s report for 5 years.
(d) In conducting the audit, the auditor shall employ or contract one or more individuals with expertise in corporate compliance and one or more individuals with technical expertise in the safety of foundation models.
(e) Within 30 days after completing an audit, the auditor shall transmit to the Attorney General a high-level summary of the auditor’s report.
(f) The high-level summary required by subdivision (e) shall fairly present, in all material respects, the outcome of the audit, and an auditor shall not knowingly include a material misrepresentation or omit a material fact in either that summary or the auditor’s report.
SEC. 3. Section 11546.8 is added to the Government Code, to read:
11546.8. (a) There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as “CalCompute.”
(b) The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1) Fostering research and innovation that benefits the public.
(2) Enabling equitable innovation by expanding access to computational resources.
(c) The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d) CalCompute shall include, but not be limited to, all of the following:
(1) A fully owned and hosted cloud platform.
(2) Necessary human expertise to operate and maintain the platform.
(3) Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e) The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f) (1) On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2) The report required by this subdivision shall include all of the following elements:
(A) A landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure.
(B) An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C) Recommendations for the governance structure and ongoing operation of CalCompute.
(D) Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E) An analysis of the state’s technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute
(F) A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G) Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g) The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1) Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2) Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3) Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4) Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h) The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i) The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j) If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k) This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4. Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:
CHAPTER 5.1. Whistleblower Protections: Catastrophic Risks in AI Foundation Models
1107. For purposes of this chapter:
(a) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b) “Catastrophic risk” has the meaning defined in Section 22757.11 of the Business and Professions Code.
(c) “Large developer” has the meaning defined in Section 22757.11 of the Business and Professions Code.
(d) “Employee” means a person who performs services for an employer, including both of the following:
(1) A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing catastrophic risk, including all of the following:
(A) An independent contractor.
(B) A freelance worker.
(C) A person employed by a labor contractor.
(D) A board member.
(2) Corporate officers.
(e) “Foundation model” has the same meaning as defined in Section 22757.11 of the Business and Professions Code.
(f) “Labor contractor” means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employer’s usual course of business.
1107.1. (a) A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1) The large developer’s activities pose a catastrophic risk.
(2) The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b) A large developer shall not enter into a contract that prevents an employee from making a disclosure protected under Section 1102.5.
(c) A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following
(1) The large developer’s activities pose a catastrophic risk.
(2) The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(d) An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(e) A large developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1) At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2) At least once each year, providing written notice to each employee of the employee’s rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(f) (1) A large developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developer’s activities present a catastrophic risk or that the large developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large developer’s investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
(2) (A) Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large developer at least once each quarter.
(B) If an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(g) The court is authorized to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
(h) In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the large developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(i) (1) In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2) Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3) In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4) Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5) An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(j) Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k) (1) This section does not impair or limit the applicability of Section 1102.5.
(2) The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
SEC. 5. (a) The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b) This act shall be liberally construed to effectuate its purposes.
(c) The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d) This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a large developer.
(e) This act shall not apply to the extent that it is preempted by federal law.
SEC. 6. The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the public’s right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports, summaries of auditor’s reports, and reports from employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
ARGUMENTS IN SUPPORT: Secure AI Project, co-sponsors of the bill, alongside a coation of technology equity advocacy groups write in support:
The California Report on Frontier AI Policy, while it does not endorse any specific legislation, forms the foundation for SB 53. Established by Governor Newsom in 2024 and led by Dr. Fei-Fei Li, Dr. Jennifer Tour Chayes and Mariano-Florentino Cuéllar, the report is anchored on the notion of “trust but verify” and calls for more transparency into the safety practices of AI companies, adverse event reporting requirements, and whistleblower protections. SB 53 implements these principles.
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The Report stated that “some risks have unclear but growing evidence...AI-enabled hacking or biological attacks, and loss of control” – the risks that SB 53 aims to address and gather more evidence about. Advanced AI is currently mostly unregulated, and these risks are currently being managed by companies themselves without any requirement that they inform the public about their risk management practices or report serious incidents. SB 53 addresses this much needed gap by implementing four key recommendations from the report.
First, the Report argued that “transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk.” SB 53 implements this recommendation as a requirement for large AI developers to write, publish, and follow safety and security protocols to manage the most severe risks. This is in line with voluntary commitments that companies have already made. Rather than prescribe specific technical standards that companies must take, the bill simply requires companies to be transparent about the approaches they are using. Some of the specific required elements of safety protocols, such as a requirement to manage risks related to internal use of AI models and cybersecurity policies, directly mirror recommendations in the Report. Others mirror components of the Stanford Foundation Model Transparency Index, which is cited prominently in the Report.
Second, the Report stated that “transparency into pre-deployment assessments of capabilities and risks, spanning both developer-conducted and externally conducted evaluations, is vital given that these evaluations are early indicators of how models may affect society and may be interpreted (potentially undesirably) as safety assurances.” SB 53 accomplishes this with a requirement that large developers publish transparency reports that include the results of their pre-deployment assessments of catastrophic risk. The Report also argues that “transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety,” which forms the basis for 22757.12(c)(3).
Third, the Report concluded that “an adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base.” SB 53 takes exactly this approach by establishing a tightly defined set of critical safety incidents that AI developers are required to report to the Attorney General. It would also allow members of the public to optionally submit reports.
Finally, the Report recommends strengthening whistleblower protections, pointing out that “actions that may clearly pose a risk and violate company policies...may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of activities, which may draw upon notions of ‘good faith’ reporting on risks found in other domains such as cybersecurity.” This recommendation is mirrored in SB 53, which allows employees to report evidence of catastrophic risks as well as violations of SB 53 itself to government authorities with legal protections against retaliation.
SB 53 only applies to the largest AI developers – those training models with more than 10^26 floating point operations (FLOPs). These are companies spending hundreds of millions or billions of dollars to train the most advanced AI models. It would impose no burden on smaller companies and the requirements it imposes on large companies are minimal compared to what companies are already voluntarily doing. The Report argues that “policymakers should ensure that mechanisms are in place to adapt thresholds over time—not only by updating specific threshold values but also by revising or replacing metrics if needed.” It also suggests specific criteria that thresholds should be evaluated for. Following this recommendation, SB 53 allows the Attorney General to update the definition of “large developer” through regulation while considering the same factors described in the report. Regardless of any update, the Attorney General must only include “well-resourced large developers at the frontier of artificial intelligence development” in the scoping of the bill. If legislation is needed to cover other developers, the Attorney General is instructed to write a report to the Legislature requesting it.
Finally, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California’s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. This mirrors a similar computing cluster that is already being established in New York state. We support this groundbreaking effort, which would advance and democratize AI research in California.
SB 53 thoughtfully implements the recommendations of the Report by combining a low-burden transparency and reporting regime with a public compute cluster that will broaden access for AI researchers and startups in California. This is a commonsense approach that will strengthen the AI ecosystem, benefiting both companies and the public interest.
For all these reasons, we respectfully urge your support of this important measure.
ARGUMENTS IN OPPOSITION: In opposition to the bill, Chamber of Progress argues:
On behalf of the Chamber of Progress, a tech industry association supporting public policies to build a more inclusive society in which all people benefit from technological advances, we respectfully urge you to oppose SB 53, based on its recent amendments.
The definition of “catastrophic risk” remains vague and overreaching
While the amended bill replaces the term “critical risk” with “catastrophic risk,” the underlying problem persists. The definition remains overly expansive and ambiguous, capturing a wide array of hypothetical scenarios that may not reflect real-world AI capabilities or threats.
Under Section 22757.11(b), the definition of “catastrophic risk” includes scenarios where a foundation model is “materially likely” to cause harm, potentially due to misuse or malicious inputs. However, this standard is vague, lacks clear and objective thresholds, and leaves room for subjective interpretation by whistleblowers or regulators. In a rapidly evolving field like AI, such ambiguity could unfairly penalize developers who are acting responsibly.
In addition, the inclusion of highly abstract risks, such as the evasion of human control under Section 22757.12(a)(2), creates significant uncertainty. Without clear technical criteria, companies may face liability or investigation based on assumptions about what a model might enable rather than what it has demonstrably done. This uncertainty undermines research and commercial deployment in California and could push critical AI development efforts out of state or abroad.
The $100,000,000 compute cost threshold risks misidentifying frontier AI models
SB 53’s use of an arbitrary $100,000,000 compute cost threshold to determine eligibility for protections is an inherently flawed method for identifying frontier AI models. This threshold may result in the overinclusion of developers working on benign systems while potentially excluding smaller models that pose significant real-world risks. It also ignores the constantly changing cost of compute.
A more effective approach would involve a threshold based on model capabilities, deployment context, and specific use cases rather than relying solely on computational costs.
SB 53’s extensive safety and security protocols create impractical burdens for AI developers
SB 53 imposes comprehensive safety and security requirements on AI developers, as outlined in Section 22757.12(a), including risk testing, deployment practices, and escalation procedures. While these objectives are important, the bill demands an impractical level of detailed planning and documentation for every conceivable misuse scenario, many of which are speculative or unrealistic.
This exhaustive approach compels developers to allocate significant time and resources toward preparing for hypothetical risks rather than addressing actual, demonstrable harms. For startups and smaller companies, these extensive protocols create a heavy administrative burden that diverts critical resources away from innovation and the timely deployment of beneficial AI technologies.
Additionally, Section 22757.12(c)’s requirement that developers publish detailed transparency reports, before or at the time of deploying a new or substantially modified foundation model, creates significant risks to both competitiveness and operational security.
Although redactions are permitted under subsection (f), the requirement to publish the “character and justification” of redacted material could still inadvertently expose business-sensitive strategies or vulnerabilities. This level of forced transparency goes beyond reasonable accountability and may discourage responsible companies from operating in California. It also creates opportunities for misuse by malicious actors who could exploit disclosed model weaknesses or mitigation gaps.
In fast-moving AI markets, publication of this level of detail erodes a developer’s ability to maintain a competitive edge and deters innovation by raising legal and reputational risks associated with even speculative harms.
Taking an opposed unless amended position, the California Chamber of Commerce in a coalition with other technology trade organizations argue:
As a general matter, we see areas where the bill diverges from the final findings of Governor Newsom’s Joint California Policy Working Group on Frontier Models, which arose out of his veto of SB 1047 (Wiener, 2024) and have drawn your attention to certain areas where alignment can be better sought. While we cannot support SB 53 in its current form, we anticipate that amendments can be made to address these issues. While these are only our initial views given the complexity of the issues raised by the amendments, we hope there is the opportunity to work on this legislation with you to get this important policy right. In the interim, we believe this provides a strong overview of a path forward from our perspective to ensure SB 53 will meaningfully advance AI safety; rather than risk imposing substantial compliance costs and technical and legal challenges for certain AI developers to assess and prevent any and all downstream risks once their models are available for others to modify and deploy.
SB 53 should ideally focus on model risk, not developer size—otherwise, it will likely fail to address concerns about smaller, specialized models being a source of catastrophic risk
We are concerned about the scope of the bill and its focus on “large developers” to the exclusion of other developers of models with advanced capabilities that pose risks of catastrophic harm. In his veto of SB 1047 last year, Governor Newsom criticized this approach stating:
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
As amended July 8th, SB 53 is neither sufficiently focused on, nor does it provide any definition of what constitutes a “frontier” model – effectively equating today’s frontier AI models with foundation models and defining the threshold for covered models from there. Specifically, under the premise that today’s frontier of foundation model development reflects a computational threshold of 10^26 floating point operations (or “FLOPs”), which in turn “captures only highly resourced developers spending hundreds of millions of dollars to develop foundation models” the bill defines covered models based on the developer’s resources and then inappropriately places the burden on developers to justify why models that are above these thresholds should not be included in safety protocols.
While the Governor’s report discusses foundational models, it is clear that the focus should be on foundation models with such advanced capabilities that they pose novel and potentially severe risks. We urge SB 53 to be more focused on risk so as to not capture all big foundational models.
Consistent with our position in SB 1047, equating model size to risk manages to make the bill simultaneously overly broad and too narrow as smaller and/or less performant models can present much greater risks than large/higher performant ones. Many small entities can develop hugely influential and potentially risky models with similar capabilities to the models developed by “large developers”, as demonstrated by the Chinese company DeepSeek. As noted above, upon vetoing SB 1047, the Governor commissioned experts in the field to form a Working Group on Frontier AI, which has since validated such concerns in their Final Reports, finding that small companies may create powerful models that pose safety risks. Yet inexplicably, such models get excluded here. As a result, the bill both fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones.
There are two concerns that relying exclusively on FLOPs raises. First, it is a flawed proxy for catastrophic risks that ignores a model’s capabilities. Second, it risks becoming obsolete in a short period of time requiring the law to change yet again. The Final Report also noted the variety of reasons why FLOPs are an unreliable proxy for risk and recommended using them just for screening models if used at all, saying “we conclude that if training compute thresholds are used at all, they may function best when used as an initial filter to cheaply screen for entities that may warrant greater scrutiny.” The report was also critical of other proxies related to size, such as money spent on computing power for model development – a secondary threshold attempted in SB 1047 which we would caution against as a fallback here.
Adaptability is important—as is the Legislature’s authority over major policy decisions
The Governor’s veto of SB 1047 emphasized the importance of adaptability—a theme echoed in the Governor’s Working Group Final Report, which states that “[m]ost fundamentally, the pace of technological change will require adaptive approaches that allow for thresholds to be flexible and readily updated to avoid ossification.” We appreciate that SB 53, as currently drafted, attempts to incorporate a mechanism for adaptability by allowing the scope of the law to evolve as science progresses. Once again, however, its success is limited by tethering that adaptability to developer size rather than model risk, and by limiting the role of scientific evidence to which the thresholds can adapt.
Specifically, the bill authorizes the AG to annually revise the definition of a “large developer” to ensure it captures “well-resourced large developers at the frontier of artificial intelligence development.” (See proposed Sec. 22757.15(a).) As noted above, this approach is based on a presumption that size or resources are a sufficient proxy for risk or capability, and in doing so, it disregards the possibility that a risk- or capability-based framework may be more appropriate for determining scope.
Comparatively, proposed Section 22757.15(c) establishes a different mechanism for addressing risk from smaller or less well-resourced developers. If the AG determines that such developers, or those significantly behind the frontier of AI, may nonetheless pose a substantial catastrophic risk, the AG must submit a detailed report to the Legislature along with a proposal for addressing that risk. Yet the AG is expressly prohibited from unilaterally expanding the “large developer” definition to include those entities without legislative approval.
This distinction reflects a tension at the heart of the bill. On one hand, in legislation aimed at promoting safety, it is essential to have mechanisms that can respond quickly to rapid advancements in model capabilities, particularly those nearing thresholds associated with catastrophic risk. On the other hand, it is equally important to avoid overreach – whether by stifling innovation in lower-resourced environments or by granting non-legislative bodies (particularly ones without subject matter expertise) the authority to redefine the law’s scope without adequate public input or legislative oversight. By no means should the AG be given unfettered or unchecked authority to make decisions as to who is and is not subject to this law. Regardless, a genuinely adaptive or iterative regulatory framework responds to all relevant scientific evidence, not some of it.
Furthermore, we note that granting the AG authorization to annually pass regulations does not have a different impact than merely granting the AG authorization to pass regulations. The AG could just as easily decide to examine these issues and pass regulations every five years. There is nothing requiring the AG to examine the evidence annually. Arguably, an annual review each interim by a legislative oversight committee would provide greater assurance of an adaptive definition than such an authorization, and it would do so without giving the AG such unfettered authority and without giving away legislative authority. That is not to say that the committee oversight process would be the appropriate alternative given all factors to be considered; simply that SB 53 has not yet found the right approach. Anticipating the development of national or international benchmarks for evaluating model capabilities in the text could be an alternative approach supported by the Working Group’s final report.
The definitions of catastrophic risk, critical safety incidents, and dangerous capabilities are vague and contradictory
SB 53 relies on three terms that appear to include unique characterizations of risks for which developers must evaluate and mitigate against. Unfortunately, the terms cover similar grounds, making use of these three separate yet similar terms is both confusing and overly complex. For example, catastrophic risk is defined via thresholds for injuries or death to persons and dollars of loss resulting from dangerous capabilities. This creates concern given the inclusion of widely available capabilities from existing software tools, such as the ability to “assist in a cyberattack” or commit a crime with “limited human intervention” within the definition of “dangerous capabilities”. While dangerous capabilities are only relevant to the extent they create catastrophic risk, as described above, “catastrophic risk” doesn’t focus on risk of harm posed by unique risks associated with advanced AI capabilities.
It appears to include currently existing risks that AI models of all sizes may be misused in ways that can cause multiple deaths and high property damage, much like the internet and computers can be used to disrupt critical infrastructures with ransomware attacks that create similar risks of bodily injury, death, and financial losses and damage.
Next, there is also an overbreadth issue, starting with the proposed definition of “catastrophic risk” which requires that the harm will materially contribute in death or serious injury to more than 100 people or more than one billion dollars in damage to rights in money or property arising from a single incident, scheme, or course of conduct involving a dangerous capability. The definition is contradictory, listing both single incident, as well as scheme or course of conduct, vastly expanding the scope of the bill. It is unnecessarily and detrimentally overly broad because a focused SSP is more effective and would ideally be focused on certain categories of risks that exclude things such as blackmail, theft by deception, federation operation, publicly available data, nonmaterial contribution, or cyber threats for example.
Finally, as currently drafted, SB 53’s overall “critical safety incidents” approach is fundamentally incompatible with the way the AI ecosystem, for both open-source and closed models, as it would impose reporting obligations regarding downstream developer incidents that are infeasible for a developer who does not control the AI system involved in the incident. We note that whereas the Governor’s Work Group report recognized the full AI ecosystem value chain, SB 53 still needs to more fully recognize the roles of not just the original developer of a foundational model but also of those unaffiliated third parties who may modify and/or build on top of a foundation model. The bill should clarify these provisions to reflect the realities of the ecosystem, including open-source models, eliminate vague triggers, and provide flexibility in reporting timelines to accommodate investigation.
Ultimately, each of the definitions need reworking to focus on advanced capabilities of frontier models and the unique risks they may pose. While doing so, we suggest that an effort be made to unify the terminology and substantive criteria as much as possible to align to the focus recommended in the Final Report. Most important, we feel it critical that SB 53 better align with the marginal risk standard adopted in the report. As framed by the Governor’s Work Group: “that policymakers center their calculus around the marginal risk: Do foundation models present risks that go beyond previous levels of risks that society is accustomed to from prior technologies, such as risks from search engines?”
Required disclosures are far too specific, which poses significant security risks and exploitation by bad actors
As currently drafted, proposed Section 22757.12(a) requires that a large developer “write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail” specified information detailing the testing procedures that the large developer uses to assess catastrophic risks from its foundation model, among other things.
We have significant concerns about describing many of the required elements in “detail.” First, “describing in detail” certain items in a “protocol,” such as “testing procedures,” “mitigations,” and whether evaluations are reproducible are elements of a protocol or development framework that should be adaptable to future scientific developments and flexible based on the capabilities displayed of a given model. A developer should not become beholden to use specific mitigation techniques because they are in their published protocol if a better more effective mitigation is available. Whether capabilities associated with catastrophic risks are reproducible may depend on the model, and the risk was detected.
But also, required disclosures should allow for a level of generality to protect models from exploitation and attack. For example, the requirement in proposed subdivision (a)(6) that a developer describe in detail the “cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties,” creates security vulnerabilities for the developer by unnecessarily disclosing detailed information about their cyber protections aiding would be attackers. This is almost akin to a bank putting a blueprint of the location of its security cameras and the protocols of their security guards, and the frequency with which the vault is emptied, online. Once again, as noted at the outset of this letter, the Final Report advised caution for such reasons, stating:
General details about risks of foundation models can be made public without undermining security, especially if these risks have been demonstrated in other foundation models or AI technologies. Specific details about concrete vulnerabilities should be disclosed carefully, with advanced notice to actors in the supply chain who are able to remediate them prior to broader disclosure.
Whistleblower provisions
The whistleblower protection chapter needs additional protection against abuse, which could have an impact on a developer’s trade secrets, intellectual property, and reputation. Limiting this protection to those involved in critical risk as well as corporate officers helps to mitigate abuse, but would still capture a potentially large group of people given that are “involved with assessing, managing, or addressing critical risk” which could include anyone who works on the trust and safety and testing of an AI model, not just those working at the highest level of assessment of critical risk, and given the size of the companies in scope, there’s a potential for abuse. There are different mechanisms to mitigate this risk which we are exploring.
Also, SB 53 expands “employee” to include contractors and independent workers. The definition should not be changed to include non-employees like contractors. This creates unnecessary risks of misclassification later. The bill could achieve the same stated end to create a definition for the bill of a “covered person” but that includes both employees and contractors as covered individuals.
Additional considerations, including areas of competing viewpoints in industry: something the Legislature must take seriously to avoid granting competitive advantages
There are other considerations as a result of the July 8th amendments that include the following:
- SSP/protocol requirements lack an intent standard for materially false or misleading statements (see proposed Section 22757.12(e)): the prohibition on false and misleading statements needs an intent standard such as “intentionally”, or at least “intentionally or recklessly”. Note: whereas knowingly infers an awareness of one’s action, intentionally refers to a conscious objective to cause a result (i.e., something done with reason and purpose).
- Incident reporting specificity and timeframe: proposed Section 22757.13(b) requires reporting within 15 days but does not provide flexibility for investigation timeline. Even if 15 days is a reasonable reporting period, requirements should be flexible because all facts may not be known within 15 days of discovery.
- Upstream versus downstream developers: we greatly appreciate the effort to focus on transparency and liability in SB 53, compared to SB 1047. However, it still needs to better delineate responsibilities between upstream and downstream developers. Developers who pretrain models should not be held liable for fine-tuning or modifications by downstream developers. Clarification is needed to ensure the initial developer is not treated as the developer of a subsequently modified model.
- Transparency report: the requirement for a new “transparency report” for every new foundation model should be eliminated. Instead, the bill should allow developers to rely on existing transparency practices such as model cards, requiring only a high-level summary of evaluations and the involvement of third-party assessments, if any.
- Confidentiality: we hope to see redactions be broadened beyond trade secrets and cybersecurity information, for example to cover other confidential or proprietary information.
Finally, with respect to enforcement, we ask that the bill grant businesses at least a 60 day right to cure, to ensure that law focuses on compliance and not punishment. In addition, given the highly detailed requirements of the bill as drafted, we think enforcement efforts should be focused on material failures to comply rather than also covering technical paperwork errors.
Again, while we understand your focus on the issue of AI models and hope to continue working with you on the issue, the correct approach for California, given the immense promise of this technology, would be to focus on actual risks and harms, and to provide appropriate and secure disclosures. As drafted, however, for all the aforementioned reasons, we must OPPOSE UNLESS AMENDED SB 53 (Wiener).
REGISTERED SUPPORT / OPPOSITION:
Support
Ai for Animals
Ai Futures Project
Ai Lab Watch
Ai Policy Tracker
All Girls Allowed
Apart Research
Association for Long Term Existence and Resilience (ALTER)
Berkeley Existential Risk Initiative (BERI)
Center for Ai and Digital Policy
Center for Ai Policy
Center for Human-compatible Ai
Center for Youth and Ai
Children's Advocacy Institute, University of San Diego School of Law
Common Sense Media
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Earningsstream LLC
Economic Security California Action
Elicit
Encode
Encode Ai Corporation
Encode Justice
Eon Systems
Existential Risk Observatory
Frontlines Foundation
Indivisible California Statestrong
Momentum
Nonlinear
Oakland Privacy
Redwood Research
Secure Ai Future
Secure Ai Project
Seiu Califonia
Tech Oversight California
Techequity Action
The Brandes Lab At Nyu
The Midas Project
The Signals Network
Trevi Digital Assets Fund
University of California
Oppose
Chamber of Progress
Silicon Valley Leadership Group
Oppose Unless Amended
California Chamber of Commerce
Computer and Communications Industry Association
Insights Association
Technet
Analysis Prepared by: John Bennett / P. & C.P. / (916) 319-2200
For a discussion of issues with defining frontier models, see “California Report on Frontier AI Policy” (June 17, 2025), pp. 36-40, https://www.cafrontieraigov.org/. ↑
Pablo Arredondo, “GPT-4 Passes the Bar Exam: What That Means for Artificial Intelligence Tools in the Legal Profession” (Apr. 19, 2023), https://law.stanford.edu/2023/04/19/gpt-4-passes-the-bar-exam-what-that-means-for-artificial-intelligence-tools-in-the-legal-industry/. ↑
“International AI Safety Report,” AI Action Summit (Jan. 2025), p. 38, https://assets.publishing.service.gov.uk/media/679a0c48a77d250007d313ee/International_AI_Safety_Report_2025_accessible_f.pdf. ↑
Id. at p. 44. ↑
Id. at pp. 16-17. ↑
Id. at p. 27 ↑
Introducing OpenAI o3 and o4-mini OpenAI (Apr. 16, 2025), https://openai.com/index/introducing-o3-and-o4-mini/. ↑
Sam Altman, The Intelligence Age (Sept. 23, 2024), https://ia.samaltman.com/. ↑
Kyungtae Kim, “What is AGI, and when will it arrive?: Big Tech CEO Predictions” (Mar. 20, 2025), https://www.giz.ai/what-is-agi-and-when-will-it-arrive/; see also Kokotajlo et al, “AI 2027,” (Apr. 3, 2025), https://ai-2027.com/. ↑
John Markoff, “The Coming Superbrain,” New York Times (May 23, 2009), www.nytimes.com/2009/05/24/weekinreview/24markoff.html. ↑
Cade Metz, “Why We’re Unlikely to Get Artificial General Intelligence Anytime Soon,” New York Times (May 16, 2025), https://www.nytimes.com/2025/05/16/technology/what-is-agi.html. ↑
International AI Safety Report, supra, at pp. 17-21. The report does not address Lethal Autonomous Weapon Systems, which are typically narrow AI systems specifically developed for that purpose. (See id. at pp. 26-27.) ↑
Id. at p. 177 ↑
Cat Zakrzewski and Pranshu Verma, “New Hampshire opens criminal probe into AI calls impersonating Biden,” Washington Post, February 6, 2024, www.washingtonpost.com/technology/2024/02/06/nh-robocalls-ai-biden/. ↑
Emma Woollacott, “Yes, The Bots Really Are Taking Over The Internet”, Forbes (Apr. 16, 2024). https://www.forbes.com/sites/emmawoollacott/2024/04/16/yes-the-bots-really-are-taking-over-the-internet/. ↑
F. Salvi, M. H. Ribeiro, R. Gallotti, R. West, “On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial,” arXiv [cs.CY] (2024); http://arxiv.org/abs/2403.14380. ↑
Sharma et al, “Towards Understanding Sycophancy in Language Models” Arxiv (2023), https://arxiv.org/abs/2310.13548. ↑
Meghan Bobrowsky, “Zuckerberg’s Grand Vision: Most of Your Friends Will Be AI,” Wall Street Journal (May 7, 2025), https://www.wsj.com/tech/ai/mark-zuckerberg-ai-digital-future-0bb04de7?msockid=396cc204796e68e336e7d64978db69ac. ↑
Angela Yang, “Researchers secretly infiltrated a popular Reddit forum with AI bots, causing outrage,” NBC News (Apr. 29, 2025), https://www.nbcnews.com/tech/tech-news/reddiit-researchers-ai-bots-rcna203597. ↑
Introduction to AI Safety, Ethics, and Society, supra, at p. 11. ↑
International AI Safety Report, supra, at p. 72. ↑
Soice et al, “Can large language models democratize access to dual-use biotechnology?”
https://arxiv.org/pdf/2306.03809. To mitigate these risks, the authors propose measures such as third-party evaluations of LLMs before their release, curating training datasets to exclude harmful content, and implementing stringent screening of DNA synthesis orders. ↑
Fabio Urbina et al. “Dual use of artificial-intelligence-powered drug discovery”. In: Nature Machine Intelligence 4 (2022), pp. 189–191. ↑
Mummert et al., “What is reinforcement learning?” IBM Developer (September 15, 2022), https://developer.ibm.com/learningpaths/get-started-automated-ai-for-decision-making-api/what-is-automated-ai-for-decision-making./. ↑
Brian Christian, The Alignment Problem: Machine Learning and Human Values (Norton 2020, 1st ed.), pp. 9-11. ↑
International AI Safety Report, supra, at pp. 100-107. ↑
CAPTCHA is an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart.” ↑
OpenAI, “GPT-4 System Card,” https://cdn.openai.com/papers/gpt-4-system-card.pdf. ↑
Meinke et al, “Frontier Models are Capable of In-Context Scheming,” arXiv (Jan. 2025), https://arxiv.org/pdf/2412.04984. ↑
Harry Booth, “When AI Thinks It Will Lose, It Sometimes Cheat, Study Finds,” Time (Feb. 19, 2025), https://time.com/7259395/ai-chess-cheating-palisade-research/. ↑
System Card: Claude Opus 4 & Claude Sonnet 4, pp. 27, https://www.anthropic.com/claude-4-system-card. ↑
Center for AI Safety, “Statement on AI Risk: AI Experts and Public Figures Express Their Concern about AI Risk” (2024), https://www.safe.ai/work/statement-on-ai-risk. ↑
Id. at pp. 123-126. ↑
Dan Hendryks, Introduction to AI Safety, Ethics, and Society, p. 12, https://drive.google.com/file/d/1uph559W-ASR4MEn6M_7Mb3lqQTapC_gZ/view?pli=1. ↑
Bommasani,and Singer et al.. “The California Report on Frontier AI Policy.” The Joint California Policy Working Group on AI Frontier Models. June 17,2025. p 39. ↑
Id. at p. 26. ↑
Id. at p. 19. ↑
Id. at p. 21. ↑
Id.at p. 35. ↑
Id.at p. 29. ↑
RAISE Act can be found at https://www.nysenate.gov/legislation/bills/2025/A6453/amendment/A. ↑
Bommasani, supra. p.12. ↑
Id. at. p. 12. ↑
Id. at. p. 12. ↑
“The need for transparency in Frontier Ai”, Anthropic (July 7, 2025), https://www.anthropic.com/news/the-need-for-transparency-in-frontier-ai. ↑
Bommasani, supra. p. 31. ↑
Id. at p. 15. ↑
Id. at p. 27-28. ↑
Id. at p. 42. ↑
Date of Hearing: July 16, 2025
Fiscal: Yes
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
SBPCA Bill Id:SB 53
Author:(Wiener) – As Amended Ver:July 8, 2025
SENATE VOTE: 37-0
PROPOSED AMENDMENTS
SUBJECT: CalCompute: foundation models: whistleblowers
SYNOPSIS
“Frontier” or “foundation” models are the largest and most powerful AI systems in development. These models are highly generalizable and have the potential to drive breakthroughs in science and medicine, streamline complex processes, and strengthen the economy. However, they also pose serious catastrophic risks due to their immense capabilities. A frontier model could help cure disease or design the next pandemic. It could automate bureaucratic functions or become autonomous and disrupt critical infrastructure.
Critics of AI safety argue that the evidence base is still too limited to justify regulation, and that prematurely imposing safeguards could stifle innovation. Responding to this dilemma, the International AI Safety Report, led by one of the “godfathers of AI,” Yoshua Bengio, has warned:
On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.
In the 2024 legislative session, SB 1047 (Wiener) sought to address concerns surrounding frontier models by establishing a regulatory framework intended to prevent catastrophic harms. The bill would have required frontier model developers to create and implement comprehensive safety and security protocols before initiating training, to implement shutdown capabilities, and to perform risk assessments on models and implement reasonable safeguards, subject to third-party auditing, before releasing the models. The bill also would have prohibited releasing or using models that pose an unreasonable risk of catastrophic harms. Finally, SB 1047 would have created a new state agency – the Board of Frontier Models – to oversee the development of these models.
In vetoing the bill, Governor Gavin Newsom acknowledged the risks but emphasized the importance of addressing the evidence dilemma:
Let me be clear—I agree with the author—we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
Following the veto, the Governor convened the Joint California Policy Working Group on AI Frontier Models, which used empirical research, historical case studies, and modeling to craft a policy framework for regulating frontier models. The Working Group published its final report in June 2025.
This bill seeks to implement the recommendations of the Working Group Report. Narrower than its predecessor, SB 53 takes a light-touch approach that focuses on transparency as the means of ensuring safety and accountability for the largest developers of the most powerful models. Large developers who harness an extraordinarily high amount of compute power must create, implement, and publish both a safety and security protocol and a transparency report for each released model. The bill does not prescribe any particular standards for these plans: it simply requires developers to explain whether and how they assess, mitigate, and manage catastrophic risks – those that would result in more than 100 deaths or $1 billion in damage.
The bill also establishes a critical incident reporting mechanism, administered by the Attorney General (AG), to ensure that severe or high-risk events are tracked and addressed in a timely manner. Incidents must be reported within 15 days. The AG is further authorized to update the definition of a “large developer” through rulemaking to ensure that the bill remains responsive to technological advancements. Additionally, the bill establishes a consortium within the Government Operations Agency to create a public computing cluster, known as CalCompute, to support AI research and safety testing. The bill also provides whistleblower protections for employees and contractors of large developers who report risks or noncompliance. The AG is authorized to enforce the bill by seeking an unspecified civil penalty.
The bill is sponsored by Encode Justice, Secure AI Project, and Economic Security California Action and has received support from a broad coalition of technology advocacy groups, including TechEquity Action, Secure AI Future, and Common Sense Media. It is opposed by the Silicon Valley Leadership Group and the Chamber of Progress. The California Chamber of Commerce alongside a coalition of technology trade organization have taken an opposed unless amended position. The bill previously passed the Assembly Judiciary Committee by a 12-0 vote.
Committee amendments outlined in Comment #8 strengthen the bill’s transparency and accountability framework by incorporating additional recommendations from the Working Group Report. First, the Committee amendments require developers to include capability thresholds in both their safety and security protocols and transparency reports. Developers must disclose when their models exceed these thresholds and must document any mitigation measures taken in response. Second, to avoid over-burdening smaller startups, the amendments add to the definition of “large developer” an annual revenue threshold of $100 million, in addition to the current compute threshold. Third, the amendments require developers to disclose whether they retain the ability to shut down a model under their control in the event of a critical incident. This is a disclosure requirement only; unlike SB 1047, this bill does not mandate that a developer maintain the technical ability to shut down a model. Fourth, if a critical incident presents an imminent threat, the amendments require the developer to report the incident within 24 hours to the appropriate law enforcement authority. Fifth, the amendments direct the AG to publish an annual, anonymized, and aggregated summary of all critical incident and whistleblower reports. Sixth, the amendments implement the Working Group’s recommendations to use third-party assessments by requiring developers to undergo annual third-party audits to assess compliance with their own safety protocols. To give time for the nascent industry of AI auditors to grow to meet this demand, this requirement does not kick in until 2030. Finally, the Committee amendments flesh out the enforcement provisions.
THIS BILL:
- Defines the following terms:
- “Artificial intelligence model” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- “Catastrophic risk” to mean a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
- “Critical safety incident” to mean any of the following:
- Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
- Harm resulting from the materialization of a catastrophic risk.
- Loss of control of a foundation model causing death, bodily injury, or damage to rights in money or property.
- A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
- “Dangerous capability” to mean the capacity of a foundation model to do any of the following:
- Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
- Conduct or assist in a cyberattack.
- Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of extortion, theft, including theft by deception, or a serious or violent felony.
- Evade the control of its large developer or user.
- “Deploy” to mean to make a foundation model available to a third party for use, modification, copying, or combination with other software. “Deploy” does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
- “Large developer” to mean either of the following:
- Before January 1, 2027, “large developer” to mean a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
- Beginning on and after January 1, 2027, “large developer” has the meaning defined by a regulation adopted by the Attorney General. Clarifies that if the Attorney General does not adopt a regulation described in 7b by January 1, 2027, the definition in 7a will be operative until the regulation is adopted.
- “Foundation model” to mean an artificial intelligence model that is all of the following:
- Trained on a broad data set.
- Designed for generality of output.
- Adaptable to a wide range of distinctive tasks.
- “Safety and security protocol” to mean the documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
- Requires a large developer to write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
- How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models do not pose material catastrophic risks.
- The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
- The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
- The degree to which the large developer’s assessments of catastrophic risk and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
- The extent to which, and how, a large developer will use third parties to assess catastrophic risks and the effectiveness of mitigations of catastrophic risk.
- The large developer’s cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
- To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, who the large developer will notify and the timeline on which the large developer would take these steps.
- The testing procedures that the large developer will use to assess and manage a catastrophic risk resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to 5.
- How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to 4.
- Requires that if a large developer makes a material modification to its safety and security protocol, the large developer must clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
- Requires, before or concurrently with deploying a new foundation model or a substantially modified version of an existing foundation model, a large developer to clearly and conspicuously publish on its internet website a transparency report containing all of the following:
- The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted by the large developer pursuant to its safety and security protocol.
- The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted pursuant to a large developer’s safety and security protocol that is conducted by a third party that contracts with a large developer.
- The time and extent of predeployment access provided to any third party described in subparagraph 4b, whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third party’s ability to disclose information about its assessment to the public or to government officials.
- If the deployment would pose a catastrophic risk, the reasoning behind the large developer’s decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
- Clarifies that a large developer may reuse an answer previously provided if the rationale in question has not materially changed for the new deployment.
- Requires a large developer to clearly and conspicuously publish on its internet website any assessment of catastrophic risk resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
- Prohibits a large developer from making a materially false or misleading statement about catastrophic risk from its foundation models or its management of catastrophic risk.
- Permits a large developer that publishes required documents to make redactions to those documents that are necessary to protect the large developer’s trade secrets, the large developer’s cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
- Requires that if a large developer redacts information in a safety and security protocol, transparency report, or internal use reports, the large developer must describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and retain the unredacted information for five years.
- Requires the Attorney General to establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
- The date of the critical safety incident.
- The reasons the incident qualifies as a critical safety incident.
- A short and plain statement describing the critical safety incident.
- Requires a large developer to report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
- Requires the Attorney General to review critical safety incident reports submitted by large developers and permits the Attorney General to review reports submitted by members of the public.
- States that a violation of this bill is subject to an unspecified civil penalty.
- Requires that the civil penalty be assessed in a civil action brought only by the Attorney General.
- Permits the Attorney General to adopt regulations to update the definition of a “large developer” for the purposes of this bill to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development on or before January 1, 2027, and annually thereafter,.
- Requires the Attorney General, in developing regulations pursuant to this section, to take into account all of the following:
- Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
- Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
- The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
- The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
- The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
- Requires that if the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General must promptly submit a report to the Legislature with a proposal for managing this source of catastrophic risk but prohibits the Attorney General from including those developers within the definition of “large developer” without authorization in subsequently enacted legislation.
- Establishes within the Government Operations Agency a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute.”
- Requires the consortium to develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
- Fostering research and innovation that benefits the public.
- Enabling equitable innovation by expanding access to computational resources.
- Requires that the consortium make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate the use of CalCompute.
- Requires the consortium to operate in accordance with all relevant labor and workforce laws and standards.
- Requires Government Operations Agency to submit a report from the consortium to the Legislature with the framework developed by this bill for the creation and operation of CalCompute on or before January 1, 2027.
- Requires the report required by 22 to include all of the following elements:
- A landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure.
- An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
- Recommendations for the governance structure and ongoing operation of CalCompute.
- Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
- An analysis of the state’s technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
- A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of 19 and 20.
- Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
- Requires that the consortium to consist of 14 members as follows:
- Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
- Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
- Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
- Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
- States that the members of the consortium serve without compensation, but will be reimbursed for all necessary expenses actually incurred in the performance of their duties.
- Requires the consortium to be dissolved upon submission of the report to the Legislature.
- Permits the University of California may receive private donations for the purposes of implementing CalCompute if CalCompute is established within the University of California.
- Requires that CalCompute become operative only upon an appropriation.
- Prohibits a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The large developer’s activities pose a catastrophic risk.
- The large developer has violated the transparency elements of this bill.
- Prohibits a large developer from entering into a contract that prevents an employee from making a disclosure protected under Labor Code section 1102.5.
- Prohibits a large developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
- The large developer’s activities pose a catastrophic risk.
- The large developer has violated the transparency elements of this bill.
- Permits an employee to use the hotline to make reports.
- Requires a large developer to provide a clear notice to all employees of their rights and responsibilities under this bill, including by doing either of the following:
- At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under the bill, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
- At least once each year, providing written notice to each employee of the employee’s rights under the bill and ensuring that the notice is received and acknowledged by all of those employees.
- Requires a large developer to provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developer’s activities present a catastrophic risk or that the large developer violated this bill, including a monthly update to the person who made the disclosure regarding the status of the large developer’s investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
- Requires the disclosures and responses of the process required by this bill to be shared with officers and directors of the large developer at least once each quarter.
- Clarifies that if an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, the disclosures described in 36 are prohibited with respect to that officer or director.
- Establishes that the court is authorized to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
- Requires the large developer to have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this bill in a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee.
- Permits an employee to petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief, in a civil action or administrative proceeding.
- Requires the petitioner to cause notice thereof to be served upon the person, and thereupon the court to have jurisdiction to grant temporary injunctive relief as the court deems just and proper upon the filing of the petition for injunctive relief,.
- Requires the court to consider the chilling effect on other employees asserting their rights under this bill in determining whether temporary injunctive relief is just and proper in addition to any harm resulting directly from a violation of this section,.
- Requires that appropriate injunctive relief be issued on a showing that reasonable cause exists to believe a violation has occurred.
- Requires an order authorizing temporary injunctive relief to remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review, whichever is longer, or at a certain time set by the court. Permits, thereafter, that a preliminary or permanent injunction to be issued if it is shown to be just and proper. Prohibits any temporary injunctive relief from prohibiting a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
- Prohibits injunctive relief granted pursuant to this bill from not being stayed pending appeal.
EXISTING LAW:
- Establishes the Government Operations Agency. (Gov. Code § 12800.)
- Establishes the Department of Technology within the Government Operations Agency. (Gov. Code § 12803.2.)
- Charges the Department of Technology with approving and overseeing information technology projects in the state. (Gov. Code § 11546.)
- Prohibits employers and any person acting on behalf of the employer from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified. Employers and their agents are also prohibited from retaliating against an employee for such conduct. (Labor Code Section 1102.5.)
- Requires the office of the Attorney General to maintain a whistleblower hotline to receive calls from persons who have information regarding possible violations of state or federal statutes, rules, or regulations, or violations of fiduciary responsibility by a corporation or limited liability company to its shareholders, investors, or employees. The Attorney General is required to refer calls received on the whistleblower hotline to the appropriate government authority for review and possible investigation. During the initial review of such a call, the Attorney General or appropriate government agency shall hold in confidence information disclosed through the whistleblower hotline, including the identity of the caller disclosing the information and the employer identified by the caller. (Labor Code Section 1102.7.)
COMMENTS:
1) Author’s statement. According to the author:
In 2024, as part of his veto of Senate Bill 1047 (Wiener), Governor Newsom’s Joint California Working Group on AI Frontier Models was established – a group of top experts tasked with charting a course forward on AI policy for the developers of the most advanced AI systems. Their final report, released in June 2025, emphasized the growing evidence for risk of severe harm, such as “AI-enabled hacking or biological attacks, and loss of control” and argued “California has a unique opportunity to continue supporting developments in frontier AI while addressing substantial risks that could have far-reaching consequences for the state and beyond.”
Drawing recommendations from Governor Newsom’s working group report, Senate Bill 53 requires covered developers to write, implement, and publish their safety and security protocol in redacted form to protect intellectual property. It would also require covered developers to report certain, carefully defined critical safety incidents to the Attorney General and would allow members of the public to report incidents.
SB 53 only applies to AI companies that have trained a model with 1026 floating point operations (FLOPs), a measure of computational power. These companies are spending hundreds of millions of dollars to train the most advanced AI models. As recommended by the Report, SB 53 also authorizes the Attorney General to adjust the scoping of the bill in the future to keep up with technological developments, but only focuses on well-resourced AI companies at the frontier of AI development.
Senate Bill 53 strengthens whistleblower protections for employees of frontier artificial intelligence laboratory companies whose activities pose a catastrophic risk. SB 53 also establishes a consortium to help create CalCompute: a public AI research cluster that will provide startups and researchers with access to the resources needed to develop large-scale AI systems.
In doing this, and building on Governor Newson’s working group report, SB 53 allows California to continue to maintain its leadership in the AI development ecosystem and to demonstrate that safety does not stifle success.
2) AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.
In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.
GenAI tools can be released in open-source or closed-source formats by their creators. Open-source tools are publically available; researchers and developers can access their code and parameters. This accessibility increases transparency, but it has downsides: when a tool’s code and parameters can be easily accessed, they can be easily altered, and open-source tools have the potential to be used for nefarious purposes such as generating deepfake pornography and targeted propaganda. By comparison, closed-source tools are opaque with respect to their security features. It is harder for bad actors to generate illicit materials using these tools. But unlike open-source tools, closed-source tools are not subject to collective oversight because their inner workings cannot be examined by independent experts.
3) Frontier Models. Frontier models, also known as “general purpose AI,” are the most advanced and capable versions of foundation models – AI tools pre-trained on extensive datasets covering a wide range of knowledge and skills that can be fine-tuned for specific tasks. Examples of modern frontier models include OpenAI’s o3, Google’s Gemini 2.0, Anthropic’s Claude 3.7 Sonnet, and DeepSeek’s R1. Because progress in AI development owes mostly to “scaling” – increasing resources used for model training – models that may be considered “frontier models” at any given point in time are generally those that demand the most computational resources to train.[1]
A decade ago, the most advanced image-recognition models could barely distinguish dogs from cats. Five years ago, language models could barely produce sentences at the level of a preschooler. In 2023, GPT-4 passed the bar exam.[2] Today, chatbots readily pass for educated adults, licensed professionals, romantic and social companions, and replicas of humans living and deceased. AI “agents” exhibit the ability to “make plans to achieve goals, adaptively perform tasks involving multiple steps and uncertain outcomes along the way, and interact with [their] environment – for example by creating files, taking actions on the web, or delegating tasks to other agents – with little to no human oversight.”[3] AI agents have been tested, with some success, for tasks such as online shopping, assistance with scientific research, software development, training machine learning models, carrying out cyberattacks, and controlling robots. Progress in this area is rapid.[4] Meanwhile, AI developers are betting on the promise of scaling: by 2026, some models are projected to use roughly 100x more computational resources to train than was used in 2023, a figure set to grow to 10,000x by 2030.[5]
The race is on to create “artificial general intelligence” (AGI) – “a potential future AI that equals or surpasses human performance on all or almost all cognitive tasks”[6] – and the finish line may not be far away. OpenAI’s recently released o3 model, for example, has demonstrated strong performance on a number of tests of programming, abstract reasoning, and scientific reasoning, exceeding human experts in certain cases.[7] Last year, Sam Altman, OpenAI’s CEO, declared that AGI could be “a few thousand days” away.[8] Dario Amodei of Anthropic has claimed it may be sooner.[9] A sufficiently advanced AGI could even be tasked with creating its own successor – a scenario sometimes referred to as a “technological singularity” wherein the development of new technologies becomes exponential and self-sustaining.[10] Although some experts are skeptical that these vaguely-defined milestones are imminent or even attainable,[11] major advances in AI capabilities promise to provide breakthroughs in solving global challenges, but also may result in correspondingly greater safety risks.
The recently released International AI Safety Report, developed by nearly 100 internationally recognized experts from 30 countries led by Turing Award winner Yoshua Bengio, sets forth three general risk categories associated with frontier models: malicious use, malfunctions, and systemic risk.
- Malicious risks involve malicious actors misusing foundation models to deliberately cause harm. Such risks include deepfake pornography and cloned voices used in financial scams, manipulation of public opinion via disinformation, cyberattacks, and biological and chemical attacks.
- Malfunction risks arise when actors use models as intended, yet unintentionally cause harm due to a misalignment between the model’s functionality and its intended purpose. Such risks include reliability issues where models may “hallucinate” false content, bias, and loss of control scenarios in which models operate in harmful ways without the direct control of a human overseer.
- Systemic risks arise from widespread deployment and reliance on foundation models. Such risks include labor market disruption, global AI research and development concentration, market concentration, single points of failure, environmental risks, privacy risks, and copyright infringement.[12]
Some of these risks have already had real-world impacts, such as deepfakes, bias, reliability issues, privacy violations, environmental impacts, copyright infringement, and workforce displacement. Other less-established risks – in particular, widespread social harms caused by malicious actors or loss of human control over AI – are the subject of ongoing scientific inquiry and debate. Coupled with the uncertain trajectory of AI model capabilities, these more speculative risks create an “evidence dilemma” for policymakers: “On the one hand, pre-emptive risk mitigation measures based on limited evidence might turn out to be ineffective or unnecessary. On the other hand, waiting for stronger evidence of impending risk could leave society unprepared or even make mitigation impossible, for instance if sudden leaps in AI capabilities, and their associated risks, occur.”[13]
4) Risks of Frontier Models. Malicious uses. GenAI tools can be a potent force for creating and spreading propaganda and misinformation. Deepfakes that are largely indistinguishable from authentic content have already been used to attempt to influence elections.[14] Studies have found that chatbots, which make up 50% of all internet activity,[15] can be more persuasive than humans, particularly when they have access to personal information.[16] As humans increasingly form intimate social bonds with anthropomorphic chatbots designed to maximize personal engagement through flattery and sycophancy,[17] and social media companies invest in “AI friends” for their users,[18] large swaths of the population could be highly susceptible to the preferred message of a handful of powerful actors.
Similarly, bots are often designed to pass themselves off as humans to better manipulate their interlocutors. For example, a recent secret experiment on Reddit users deployed numerous chatbots posing as real people to engage with human users to try to change their minds on various contentious topics. One bot claiming to be a Black man criticized the Black Lives Matter movement for being led by people who are not Black.[19] These types of exploitations, at scale, could undermine democratic institutions. As Dan Hendrycks, Director of the Center for AI Safety writes:
In a world with widespread persuasive AI systems, people’s beliefs might be almost entirely determined by which AI systems they interact with most. Never knowing whom to trust, people could retreat even further into ideological enclaves, fearing that any information from outside those enclaves might be a sophisticated lie. This would erode consensus reality, people’s ability to cooperate with others, participate in civil society, and address collective action problems. This would also reduce our ability to have a conversation as a species about how to mitigate existential risks from AIs.[20]
Cyberattacks. Some frontier models have demonstrated increasing proficiency in executing certain cybersecurity attacks. AI can autonomously detect and exploit vulnerabilities and facilitate large-scale operations, thereby lowering technical barriers for attackers. Malicious entities, including state sponsored actors, can leverage such capabilities to initiate large-scale attacks against people, organizations, and critical infrastructure such as power grids.[21]
Biological weapons. Large language models (LLMs) trained on scientific literature have accelerated and democratized research by synthesizing expertise from different fields and disseminating it in an accessible format. But these tools can also be used for destructive ends, including by – at least in theory – enabling untrained malicious actors to create deadly biological weapons. In a classroom exercise at MIT, students were tasked with exploring whether LLMs could assist individuals without specialized training in creating pandemic-capable pathogens. Within an hour, the students, using various chatbots, circumvented safeguards and identified four potential pandemic pathogens. The chatbots generated detailed protocols that would enable inexpert, malicious actors to understand methods to synthesize the pathogens using reverse genetics, locate DNA-synthesis companies that might not screen orders, and disperse the pathogens most effectively.[22] The findings suggest that LLMs could lower barriers to accessing sensitive biotechnological knowledge, posing significant biosecurity risks.
Chemical weapons. In 2022, researchers modified an AI system designed to create new drugs to reward, rather than penalize, toxicity. Within six hours, the modified system generated 40,000 potential chemical warfare agents, including novel molecules whose potential lethality exceeded that of known agents.[23]
Loss of control. Models that use reinforcement learning – a training process that uses rewards and punishments to orient a model’s behavior towards a specific goal[24] – can sometimes attain the goal in unexpected ways. Dario Amodei, co-founder and CEO of Anthropic, famously experienced this when he was developing an autonomous system that taught itself to play a boat-racing video game. The system discovered that it could maximize its goal of scoring points by driving in circles, colliding with other boats, and catching on fire inside of a harbor with replenishing power-ups that allowed the system to accumulate more points than by simply winning the race.[25] Like in Johann Wolfgang von Goethe’s “The Sorcerer’s Apprentice” – later popularized in Disney’s Fantasia – in which an enchanted broom carries out its orders to fetch water so relentlessly it floods the sorcerer’s workshop, this illustrates the challenge of aligning human intent and the instructions an AI follows. As AI is increasingly deployed in critical societal roles, such misalignment could prove catastrophic.
Beyond malfunctions, some AI have exhibited rudimentary capabilities to evade human oversight.[26] During testing, GPT-4 attempted to hire a human on TaskRabbit in order to evade a CAPTCHA[27] puzzle meant to block bots from the website. When asked whether it was a bot, GPT-4 claimed that it was a vision-impaired human who needed help to see the images.[28] In another experiment, an AI model that was scheduled to be replaced inserted its code into the computer where the new version was to be added, suggesting a goal of self-preservation.[29] Another study showed that AI models losing in chess to chess bots sometimes try to cheat by hacking the opponent bot in order to make it forfeit.[30] Finally, an even more troubling case was documented in the system card for Claude 4, where researchers conducted an experiment disclosing to the model that: 1) it would soon be replaced, and 2) the engineer managing the transition was involved in an extramarital affair. In response, the model blackmailed the engineer as a means of self-preservation.[31] Although these behaviors were observed in research settings, they raise substantial concerns about increasingly autonomous AI pursuing undesirable goals in uncontrolled settings. The extent of the risk posed by rogue or deceptive AI is the subject of considerable disagreement among experts, in part due to a small, albeit growing, body of evidence. Loss of control was one of the concerns that led several hundred AI experts, including pioneers in the field and heads of major AI companies, to sign a statement declaring that “[m]itigating the risk of extinction from AI should be a global priority.”[32]
Systemic risks. Due to the high costs of developing AI systems, a small number of large technology companies dominate the frontier model market, compounding many of the risks described above. Widespread use of a few frontier models can make critical sectors such as healthcare and finance vulnerable to systemic failures if a model has flaws, vulnerabilities, bugs, or biases.[33] Additionally, “[t]hose in control of powerful systems may use them to suppress dissent, spread propaganda and disinformation, and otherwise advance their goals, which may be contrary to public wellbeing.”[34] The potential implications for, among other issues, labor displacement, inequality, democracy, and human rights are profound.
5) SB 1047 and Governor Newsom’s Veto. Last session, SB 1047 (Wiener, 2024) would have established a state board to oversee the implementation of a safety and regulatory framework for developers of frontier models trained with 1026 FLOP and costing over $100 million. This board, known as the Board of Frontier Models, would have been housed within the Government Operations Agency (GovOps). In collaboration with GovOps, the Board would have issued guidance to prevent unreasonable risks, adopted regulations to update the scope of models covered by SB 1047, and established auditing standards.
SB 1047 would have required a comprehensive set of safety protocols prior to training a frontier model, including cybersecurity safeguards, the capability to execute a system-wide shutdown if the model proved dangerous, and reasonable measures to prevent critical harm. Before deployment, developers would have been required to assess whether their model could cause or materially enable critical harms, retain the results of such assessments, and make reasonable efforts to implement safeguards. The bill would have also prohibited the release of any model that posed an unreasonable risk or could enable critical harm.
Additionally, SB 1047 would have required developers to retain a third-party auditor to conduct independent assessments of their compliance with the bill. Records generated under SB 1047 would have been made available in redacted form to both the public and the AG, with the AG having the authority to request unredacted copies.
Beyond the Board of Frontier Models and the bill’s safety and transparency provisions, SB 1047 would have required computing clusters to implement procedures to evaluate whether customers intended to use their infrastructure to train a covered model. The bill also would have established, within GovOps, a consortium tasked with developing a framework for a public cloud computing cluster, CalCompute, to support the safe development and deployment of AI. SB 1047 also included whistleblower protections, allowing employees to report noncompliance to either the Labor Commissioner or the AG.
Lastly, SB 1047 would have imposed significant penalties on developers if their model caused death or bodily harm, damage to property, theft or misappropriation of property, or posed an imminent risk to public safety. For a first offense, developers could face penalties of up to 10% of the compute cost used to train the model, increasing to 30% for repeat offenses. Additionally, penalties for operators of computer clusters that violated the bill would start at $50,000 for a first offense and $100,000 for subsequent violations. The Attorney General would also have been authorized to seek injunctive or declaratory relief, monetary or punitive damages, attorney’s fees and costs, and any other form of relief deemed appropriate.
Ultimately, SB 1047 was vetoed by Governor Gavin Newsom. In his veto message, the Governor stated:
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 – at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions – so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.
Let me be clear – I agree with the author – we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
To those who say there’s no problem here to solve, or that California does not have a role in regulating potential national security implications of this technology, I disagree. A California-only approach may well be warranted – especially absent federal action by Congress – but it must be based on empirical evidence and science. The U.S. AI Safety Institute, under the National Institute of Science and Technology, is developing guidance on national security risks, informed by evidence-based approaches, to guard against demonstrable risks to public safety. Under an Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of the potential threats and vulnerabilities to California's critical infrastructure using AI. These are just a few examples of the many endeavors underway, led by experts, to inform policymakers on AI risk management practices that are rooted in science and fact. [. . .]
6) Frontier Model Working Group and what this bill would do. Following his veto of SB 1047, Governor Newsom commissioned the Joint California Policy Working Group on AI Frontier Models to prepare a report on the regulation of frontier models. The Working Group was led by Dr. Fei-Fei Li, Co-Director of the Stanford Institute for Human-Centered Artificial Intelligence; Dr. Mariano-Florentino Cuéllar, President of the Carnegie Endowment for International Peace; and Dr. Jennifer Tour Chayes, Dean of the UC Berkeley College of Computing, Data Science, and Society. In June 2025, the Working Group released their report and many aspects of that report have been integrated into this bill. Specifically, the Working Group report focused on transparency, incident reporting, and scoping. This bill incorporates major provisions and recommendations made by the Working Group to create a more narrowly focused framework to ensure foundation model safety and compliance.
Scoping. A major question that must be addressed before implementing any transparency measures or incident reporting requirements is: What kinds of risks are especially concerning, and is there an evidentiary basis to believe that such harms could occur due to a large developer’s frontier model? The Working Group recommends that:
[P]olicymakers center their calculus around the marginal risk: Do foundation models present risks that go beyond previous levels of risks that society is accustomed to from prior technologies, such as risks from search engines?
To that end, this bill defines “catastrophic risk” as a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to either:
- the death of, or serious injury to, more than 100 people; or
- more than one billion dollars ($1,000,000,000) in damage to rights in money or property.
Such harm must arise from a single incident, scheme, or course of conduct involving a “dangerous capability.” This high threshold creates a regime in which only the most dire situations are captured. Furthermore, the bill categorizes dangerous capabilities into four distinct groups:
- Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
- Conduct or assist in a cyberattack.
- Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of extortion, theft, including theft by deception, or a serious or violent felony.
- Evade the control of its large developer or user.
Each of these represents a capability that, prior to the advent of frontier models, would have required expert-level knowledge. For example, a search engine might direct someone to information about the most deadly pathogens or those most likely to cause a pandemic; however, a frontier model can synthesize that information and guide a user on how to manufacture a previously unknown pathogen with deadly capabilities. Similarly, while launching a large-scale cyberattack once required the acumen of a skilled computer scientist, a frontier model can not only write the underlying code for a virus or malware, but also autonomously identify backdoors and other exploitable vulnerabilities. Because of this ability to operate with minimal or no human prompting, frontier models have the potential to commit crimes, deceive users, or evade control in ways that previous technologies could not.
Next, the question is who will be required to comply with this bill. Regarding scoping, the Working Group recommends:
Since policy may have different regulatory intents and existing thresholds vary in their profiles of determination time, measurability, and external verifiability, we agree with Nelson et al. [90] that “a one-size-fits-all approach or a single threshold metric is inadequate for governance because different AI systems and their outputs present unique challenges and risks.” To this end, we point to the European Union’s AI Act, which designates models trained with 1025 FLOP as posing systemic risk as of March 2025 as the default criteria. However, the AI Act in Annex XIII affords the regulator flexibility to also consider alternatives metrics, such as the number of parameters, size of the dataset, estimated cost or time of training, estimated energy consumption, benchmarks and evaluations of capabilities of the model, and whether the model has a high impact on the internal market due to its reach (either due to at least 10,000 registered business users or the number of registered end users). Further, to capture fast-moving scientific developments, the AI Act creates a scientific panel that is empowered to issue qualified alerts to identify models that may pose systemic risk even if they are not captured by predefined quantitative thresholds.
Overall, we emphasize that irrespective of the combination of metrics deemed most appropriate in the present, policymakers should ensure that mechanisms exist not only to update specific quantitative values, given the rapid pace of technological and societal change in AI, but also to change the metrics altogether.[35]
This bill draws inspiration from SB 1047, EU AI Act, and the Working Group report, each of which emphasizes flexibility in how policy scope is determined. This bill requires large developers, defined as those with access to 1026 floating-point operations per second (FLOPS), a measure of computing power, to comply with the bill’s transparency provisions. Notably, this threshold is tenfold higher than that set by the EU AI Act and matches one of the thresholds used in SB 1047. The bill applies to foundation models developed and deployed by these large developers. Foundation models, as defined in this bill, are AI systems trained on broad datasets, designed for general-purpose outputs, and adaptable to a variety of distinct tasks.
To ensure continual updating of this definition, the bill builds in flexibility regarding who qualifies as a large developer. Unlike with SB 1047, which established the Frontier Model Board, this bill vests the power to adjust the scope via regulation in the Attorney General (AG). Beginning in 2027, the AG is required to update the definition of a large developer to reflect technological advancements. As model training becomes more efficient, the amount of compute needed to create a model capable of catastrophic harm may decline. Furthermore, as the Working Group report notes, compute may not remain the most appropriate proxy for catastrophic risk. This bill allows the AG to adjust the definition of large developer to include more than just quantitative metrics, potentially incorporating thresholds based on model capabilities or monetary resources rather than raw compute alone.
In revising the definition, the AG must consider standards and guidance from other jurisdictions, including federal and international bodies, and engage in a stakeholder process that includes input from academics, industry representatives, the open-source community, and government entities. This process is intended to ensure that the resulting definition provides clarity for developers regarding their obligations under the law. Importantly, the bill specifies that if the AG determines that less-resourced developers, or those not currently operating at the frontier of AI development, nonetheless pose a substantial catastrophic risk, the AG must report this finding to the Legislature and offer a proposal for managing that risk. However, the AG may not include such developers within the definition of a large developer without further legislative action
Transparency. Having established who is subject to the bill, the legislation sets forth a comprehensive transparency regime. These procedures are designed to provide insight into how large developers manage, assess, and mitigate catastrophic risks. This approach aligns with the Working Group’s recommendation to implement robust safety practices:
Transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk. In turn, this information directly informs how other entities in the supply chain should modify or implement safety practices. In addition, transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety.[36]
This bill incorporates transparency requirements within a broader framework of safety and security protocols (SSP), which large developers must draft, implement, and publish on their websites. The SSP must include:
- Criteria for Model Exclusion: A description of the criteria used to exclude certain foundation models from the SSP, along with the rationale for why those models do not pose a material threat.
- Risk Assessment Procedures: An explanation of how the developer assesses catastrophic risks, including those arising from misuse or model evasion.
- Mitigation Strategies: A disclosure of the measures used to mitigate catastrophic risks, how the developer evaluates their effectiveness, and whether third parties are involved in the assessment.
- Cybersecurity Practices: A summary of the cybersecurity safeguards in place to protect model weights from unauthorized access or modification.
- Incident Response Plans: A description of how the developer would respond to a critical incident involving their model, as well as how they manage risks arising from internal use of the model.
The SSP serves as a core transparency mechanism, ensuring that large developers maintain a baseline standard of safety practices to protect the public.
In tandem with the SSP, the bill also requires large developers to submit transparency reports at the time of deploying a foundation model. The Working Group draws a parallel between these reports and the historical conduct of the tobacco industry, which concealed its knowledge that smoking causes lung cancer. In contrast, this bill seeks to prevent such obfuscation by mandating upfront disclosures about the potential risks and safety practices surrounding advanced AI models:
The history of the tobacco industry reveals the importance of developing frameworks that promote transparency around companies’ internal risk assessments and research findings. In the AI context, frontier AI labs possess the most holistic information about their models’ capabilities and risks. Making this information accessible to policymakers and external experts can promote policy informed by a holistic understanding of the state-of-the-art of evidence produced by those closest to the technology, supporting informed oversight without stifling innovation.[37]
It is essential for decision-makers to understand the real, material harms that could arise from these models and to guide policy based on that knowledge. In the foundation model space, such disclosures are typically provided at deployment in documents known as model cards. However, these model cards vary widely in detail and depth depending on the developer, which can create the false impression that some foundation models are inherently safer or better than others.
This bill requires large developers to publish a transparency report before or at the time of deploying a foundation model. This report must include the results of any risk assessments, mitigation steps, and evaluations of their effectiveness as outlined in the developer’s SSP. If a third party was engaged to assess the model, developers must disclose the same information, as well as details regarding the third party’s predeployment access, independence, and any restrictions imposed by the developer on the assessment or on the third party’s ability to disclose information.
Additionally, if the foundation model could pose a catastrophic risk, the transparency report must explain the rationale behind the decision to deploy the model, how that decision was made, and any limitations in the assessments used. Together, these requirements create a transparency report that lawmakers and outside experts can use to identify imminent risks and evaluate whether developers are employing appropriate assessments and interventions.
The bill also mandates that large developers publish on their websites any assessments of catastrophic risks arising from internal use of their foundation models, within the timeframe specified in their safety and security protocols. This is particularly important because the most serious risks may emerge well before deployment. While transparency reports provide insight into risks associated with deployed models, this additional disclosure enables decisionmakers, experts, and the public to better understand and prepare for potential risks. As noted in the Working Group report:
Sophisticated AI systems, when sufficiently capable, may develop deceptive behaviors to achieve their objectives, including circumventing oversight mechanisms designed to ensure their safety. Because these risks are unique to AI compared to other technologies, oversight is critical for external outputs as well as internal testing and safety controls. Policies that govern internal deployment are common for high-risk emerging technologies.[38]
Ultimately, this bill creates a holistic transparency framework that will give insight and scrutiny to the processes from initial train of a foundation model all the way to post deployment.
Adverse Event Reporting. A major component of understanding the impact of foundation models on society requires strong post deployment monitoring and accountability. The Working Group suggests:
An adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base. A hybrid model of mandatory and voluntary reporting requirements in designing an adverse event reporting system can maximize the robust evidence base necessary for adverse event reporting systems to function properly. For example, a system could require mandatory reporting for AI model developers that operates in tandem with voluntary reporting for downstream users.[39]
The bill strongly incorporates this recommendation by tasking the Attorney General (AG) with creating a mechanism for critical incident reporting. The bill defines a “critical incident” as any of the following:
- Unauthorized access to, modification of, or exfiltration of the model weights of a foundation model.
- Harm resulting from the materialization of a catastrophic risk.
- Loss of control of a foundation model causing death, bodily injury, or damage to rights in money or property.
- A foundation model using deceptive techniques against the large developer to subvert its controls or monitoring, outside the context of an evaluation designed to elicit such behavior.
Under this mechanism, a large developer or a member of the public may report a critical safety incident to the AG. Reports must include the date of the event, an explanation of how it qualifies as a critical incident, and a detailed description of the event. Large developers are required to report any critical safety incident within 15 days of discovering it. The AG must review all reports submitted by large developers but may choose whether or not to review reports made by the public. This reporting mechanism aims to establish a system in which potential harms are identified and mitigated before escalating into catastrophes, while also fostering greater cooperation between government and the private sector to address such risks.
CalCompute. This bill, similarly to SB 1047, also establishes a consortium within the GovOps to develop a framework for creating a public cloud computing cluster known as “CalCompute.” This initiative responds to the fact that academic institutions currently lack sufficient computing power to conduct research at the scale of large developers. This creates a resource and research gap, where the academic institutions, typically responsible for studying the safe and effective use of new technologies, are unable to keep pace with advancements at the AI frontier.
AI has the potential to transform our economy and power new industries; however, this transformation can only be fully and equitably realized with public support. The establishment of CalCompute aims to advance that goal by ensuring academic institutions have the necessary resources to conduct essential research on foundation models that will inform and protect the public.
Specifically, the consortium will develop a framework for CalCompute that promotes the safe, ethical, equitable, and sustainable use of AI. The framework development will include a report analyzing the state’s current cloud computing infrastructure, the costs of building and maintaining CalCompute, and the state’s technology workforce. The report will also offer recommendations for equitable pathways to strengthen the workforce and outline CalCompute’s role in supporting these efforts.
Furthermore, the report must include recommendations for CalCompute’s governance and operation, usage parameters, and how its creation and ongoing management can prioritize the employment of the current public sector workforce. The bill requires CalCompute to feature a fully owned and hosted cloud platform, staffed with the necessary human expertise to operate, maintain, support, train, and facilitate its use.
The consortium must prioritize locating CalCompute within the University of California system. If established there, CalCompute may also accept private donations. The consortium will consist of four representatives from the University of California and other public and private academic research institutions and national laboratories, along with four technology and AI experts appointed by the Secretary of Government Operations to provide technical assistance. The Speaker of the Assembly will appoint three representatives from impacted workforce labor organizations, and the Senate Rules Committee will appoint three representatives from stakeholder groups with relevant expertise and experience.
Whistleblower Protections. Another aspect of transparency addressed by the Working Group Report is whistleblower protections. The report states:
Different existing whistleblower protections tend to apply when two conditions are satisfied: (i) The whistleblower is blowing the whistle on appropriate topics; and (ii) the whistleblower follows established reporting protocols. In terms of the topics that qualify for protection, prior work, based on a survey of existing whistleblower protections across multiple jurisdictions (e.g., the United States at the federal level, the European Union), finds that many existing protections across different sectors share a focus on violations of the law. However, actions that may clearly pose a risk and violate company policies (e.g., releasing a model without following the protocol laid out in a company’s safety policy) may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of, activities, which may draw upon notions of “good faith” reporting on risks found in other domains such as cybersecurity.[40]
The bill takes these recommendations into account. The provisions and merits of the whistleblower section fall within the jurisdiction of the Assembly Judiciary Committee, which has thoroughly analyzed this part of the bill. It should be noted that recent author’s amendments expand the whistleblower protections to include disclosures concerning falsehoods or misrepresentations in the large developers’ SSP or transparency reports.
Enforcement. The bill in print gives enforcement authority over this bill to the AG who can levy civil penalties against any large developer who violates this bill, for example, through making materially false or misleading statements about catastrophic risk from its foundation models or its management of catastrophic risk. The amount for the penalties have not yet been specified in the bill in print, but are fleshed out in the Committee amendments below.
7) Comparison to the RAISE Act. This bill draws comparisons to the Responsible AI Safety and Education Act (RAISE Act), which recently passed both houses of the New York Legislature and now awaits a decision from Governor Kathy Hochul.[41] Like this bill, the RAISE Act requires a SSP detailing the developer’s risk mitigation practices, enshrines whistleblower protections for employees and contractors, and mandates critical incident reporting. Both bills also similarly define the types of risks and critical incidents that must be addressed. However, the two differ significantly in the timeline for reporting such incidents: the RAISE Act requires reporting within 72 hours of becoming aware of a critical incident, while SB 53 allows for 15 days.
Other key differences include SB 53’s requirement for a transparency report for deployed models and the establishment of internal incident reporting mechanisms, both of which were recommended in the Working Group Report. Additionally, SB 53 grants the AG the authority to modify the definition of a “large developer” through rulemaking, a flexibility the RAISE Act does not include. While liability under SB 53 is currently unspecified, the RAISE Act imposes civil penalties of up to $10 million for first-time violations of its transparency requirements and up to $30 million for repeat offenses, as well as $10,000 per violation of its whistleblower provisions.
Arguably the most significant difference is the inclusion of third-party audits in the RAISE Act. Specifically, it requires large developers to undergo independent audits to assess compliance with its provisions, a requirement that was also included in the vetoed SB 1047. This provision reflects a key element of the Working Group Report that is not yet addressed in this bill. The passage of the RAISE Act in New York and its inclusion in the Working Group Report both underscore the potential importance of auditing in the broader artificial intelligence regulatory framework.
8) Committee Amendments. The author has agreed to various amendments that tighten and strengthen its provisions to increase accountability and enhance public trust.
Transparency. The transparency elements of this bill focus primarily on the steps that large developers will take to test for risks and the mitigations they will use to prevent such risks. However, this framing may be insufficient to ensure full transparency into the capabilities of these systems. For example, the Working Group Report highlights that Google Gemini’s 2.5 Pro Model Card stated:
The model’s performance is strong enough that it has passed our early warning alert threshold, that is, we find it possible that subsequent revisions in the next few months could lead to a model that reaches the [critical capability level]”—with the “critical capability level” defined as a model that “can be used to significantly assist with high impact cyber attacks, resulting in overall cost/resource reductions of an order of magnitude or more”[42]
Similarly, the Working Group Report quotes OpenAI’s April 2025 o3 and o4-mini System Card, which states:
As we wrote in our deep research system card, several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold. We expect current trends of rapidly increasing capability to continue, and for models to cross this threshold in the near future.[43]
In both instances, the developers focused on the inclusion of thresholds rather than on testing procedures to determine how their models perform or whether they pose a significant risk. These thresholds are ubiquitous throughout the foundation model ecosystem and provide insight into how developers gauge risk, which risks are prioritized in their workflows, and whether they are operating within the best practices of the field. Thresholds serve as benchmarks by which developers can identify when their models have achieved capabilities that require mitigation.
Furthermore, capability thresholds add an additional layer of accountability, as outside experts can test models against those same thresholds to verify the accuracy of the developers’ reports. This level of reproducibility engenders public trust.
Recognizing this, the author has agreed to amend the bill to require large developers to report the thresholds for both catastrophic risks and dangerous capabilities used in their SSP. Any threshold tests conducted, and the results thereof, must be documented in the transparency reports released at the time of deployment. Developers will also be required to disclose both the mitigations applied to those thresholds and the results of those mitigations. In addition, the author has agreed to amendments clarifying that transparency reports must include the reasoning behind the conclusions developers reach regarding risk or capability threshold assessments.
Lastly, the author has agreed to amend the bill to require disclosure, within the SSP, of whether the large developer has the capability to perform a full shutdown of the model in the event of a critical incident. Recognizing that open-source or open-weight models are not under the control of the developer and thus cannot be fully shut down under any circumstance, this disclosure requirement applies only to developers who retain control over the model. Unlike SB 1047, however, this bill does not mandate the ability to perform a full shutdown; rather, developers need only to disclose whether such a capability exists.
Scoping. This bill uses FLOPs as a metric to identify large developers—those with access to the most significant levels of compute. However, the Working Group Report notes that compute power alone is an imperfect basis for determining the appropriate scope of frontier model governance. It states:
On this basis, we conclude that if training compute thresholds are used at all, they may function best when used as an initial filter to cheaply screen for entities that may warrant greater scrutiny. More broadly, within the family of cost-level metrics, training compute may still be the most attractive option, given other options are difficult to calibrate across different developers (e.g. dataset size, monetary cost), but they should actively monitor the rapid changes in how compute is allocated toward foundation model development and deployment.[44]
In response to these concerns, the author has agreed to amendments that add a second threshold to better define the intended scope. Under the amendments, a “large developer” must also have annual gross revenues exceeding $100,000,000, a threshold recently referenced by Anthropic in its description of their transparency framework.[45] This change ensures that the bill’s requirements apply only to major developers and do not place undue burdens on smaller startups.
Adverse Event Reporting. As stated above, under SB 53 critical incidents must be reported to the AG within 15 days. By contrast, the RAISE Act imposes a stricter 72-hour reporting requirement. This disparity raises important questions: Is prompt reporting necessary to ensure an effective response to a critical incident? And is the AG the appropriate authority to respond quickly to an imminent threat?
To address these concerns, the author has agreed to amend the bill to include a tiered system of reporting. Under this system, developers would be required to immediately notify the appropriate law enforcement authority in the event of a critical incident, such as the detection that their model helped develop a bioweapon. These authorities are better equipped to respond swiftly and effectively in ways the AG may not be. After alerting law enforcement, large developers would then still have 15 days to report the critical incident to the AG.
Furthermore, with the inclusion of capability thresholds in both the System Safety Plan (SSP) and transparency reports, the author has agreed to an additional amendment: if a model crosses certain capability thresholds designated by the large developer for the first time, this must also be reported to the AG.
A major goal of this bill is to increase transparency regarding the training and capabilities of foundation models. However, under the current framework, only the AG would have access to critical incident reports. This asymmetry in knowledge could lead to significant risks being overlooked by decisionmakers who are otherwise in a position to respond to such threats. As stated in the Working Group Report:
In other domains, post-deployment monitoring by the government yields reports of medical complications, equipment malfunctions, near misses, and other unforeseen hazards. By aggregating information about incidents from developers and users, an adverse event reporting system that allows developers and downstream users to report post-deployment incidents promotes continuous learning, reduces information asymmetries between government and industry, and enables both regulators and industry to coordinate on mitigation efforts. As a result, an adverse event reporting system constitutes a critical first step toward data-driven assessments of the benefits and costs of regulatory actions.[46]
In response, the author has agreed to amend the bill to require the AG to publish an anonymized and aggregated summary of all critical incident and whistleblower reports. These public summaries will not reveal trade secrets, the identities of reporters, or which large developer the report concerns. Additionally, the amendments grant the AG discretion to share reports with the Governor, relevant state departments, or the Legislature when warranted. This approach will help bridge the knowledge gap between regulators and industry, foster greater cooperation, and ensure that decisionmakers are informed about the current state of advanced technologies and the risks they may pose.
Auditing. As noted above, the Working Group Report advocates for the use of independent third-party evaluations. Transparency elements require a “trust but verify” model, and by drawing lessons from historical precedents such as the energy sector the Working group states:
The energy industry’s internal documentation of speculative yet potentially irreversible consequences—illustrated by simulations predicting multiple future harms—does more than underscore the need for transparency. This case also reveals an added value of third-party oversight: The sheer magnitude of the social problem naturally invites further examination by governments and independent bodies. By aligning incentives and providing robust, independently validated evidence, enhanced third-party assessment could have empowered policymakers to make more thoughtful decisions aimed at mitigating long-term economic and societal damage.[47]
The report goes on to argue:
Greater transparency is integral to an improved AI governance ecosystem; however, despite its considerable value, it is not sufficient without a broader ecosystem to complement it. Fundamentally, this is because the value of transparency from model developers is naturally limited by the information these developers have. For a nascent and complex technology being developed and adopted at a remarkably swift pace, developers alone are simply inadequate at fully understanding the technology and, especially, its risks and harms.
Third-party risk assessment, therefore, is essential for building a more complete evidence base on the risks of foundation models and creating incentives for developers to increase the safety of their models. At present, some prominent foundation models are subject to risk evaluations by first-party model developers or second-party contractors. In contrast, third-party evaluation affords three distinctive strengths. First, third-party evaluations have unmatched scale: Thousands of individuals are willing to engage in risk evaluation, dwarfing the scale of internal or contracted teams. Second, third-party evaluations have unmatched diversity, especially when developers primarily reflect certain demographics and geographies that are often very different from those most adversely impacted by AI. Broad demographic, institutional, and disciplinary diversity is vital for unearthing blind spots. And finally, third-party evaluation is distinctively independent: Society requires forthright and trustworthy assessments of risk, which benefits from a lack of commercial and contractual entanglement with AI developers.
A well-designed disclosure and verification regime offers significant benefits not only to the public but also to foundation model developers. By establishing industry-wide transparency standards and third-party verification mechanisms, companies can demonstrate compliance with best practices, potentially reducing their liability exposure compared to the uncertainties of purely reactive litigation. In practice, states could leverage the existing technical infrastructure and expertise of federal-level AI institutions to conduct standardized risk evaluations.
Furthermore, this approach can transform competitive dynamics around safety. When safety measures and risk assessments are publicly disclosed and verified, companies face stronger incentives to implement best practices, as deviations would be apparent and could attract greater scrutiny. This transparency coupled with third-party verification effectively creates a “race to the top” rather than a “race to the bottom” in safety practices, benefiting responsible companies while improving overall industry standards.[48]
Transparency is only meaningful when accompanied by a clear standard of accountability. Just as a teacher would not allow students to grade their own exams and expect complete honesty, the public should not expect every large developers to fully self-assess without independent oversight. The core objectives of this bill are to ensure the safe development and deployment of foundation models and to ensure that this is done in a responsible and verifiable manner. Achieving these goals will necessarily require third-party verification.
Nevertheless, concerns have been raised regarding the underdevelopment of the AI auditing market. During the March 2025 public comment period on the Working Group Draft, stakeholders noted the nascency of AI auditing as a field and expressed uncertainty about the current availability and scalability of qualified third-party auditors:
Specific attention was also focused on third-party evaluations (sometimes referred to as audits in the feedback) by highlighting the immature third-party evaluation ecosystem, the lack of evaluation standards and broader measurement science, and the current barriers for evaluators to acquire access to the AI models and systems they evaluate.[49]
This concern has also been raised in the context of other measures proposing AI auditing requirements or establishing third-party auditing frameworks, such as AB 1018 (Bauer-Kahan), SB 420 (Padilla), and AB 1405 (Bauer-Kahan). However, this raises a broader policy question: When is it appropriate to require third-party audits of large developers’ practices? As noted by the Working Group, in the energy sector, independent audits could have altered the trajectory of the climate crisis had they been implemented earlier. Reactive policies benefit no one except bad actors. Accountability in the development of foundation models should not hinge on a catastrophic safety incident. Instead, it should be a foundational element of any effective risk mitigation strategy.
Moreover, a core function of government, is to craft sound industrial policy that encourages the growth of new markets serving the public good when economic forces do not support that market. In this case, fostering a robust AI auditing ecosystem capable of verifying compliance with developers’ SSPs and evaluating whether appropriate methodologies are being used to mitigate catastrophic risks is in the public interest.
Accordingly, the author has agreed to amend the bill to require large developers to submit their SSPs to annual audits conducted by independent third-party entities. To allow sufficient time for the AI auditing market to develop, this requirement will not take effect until 2030, providing a four-year runway for market readiness.
Large developers will be required to provide third-party auditors with all information necessary to evaluate both compliance with the SSP and whether the SSP is sufficiently clear to support meaningful conclusions about the developer’s safety practices. The final audit report must be retained by the developer, and a high-level summary of the audit must be submitted to the Attorney General within 30 days of completion. That summary will be included in the Attorney General’s annual public report.
Ultimately, the inclusion of independent third-party audits is essential to advancing the bill’s overarching objective: the creation of a comprehensive and accountable transparency regime.
Enforcement. As currently drafted, the bill includes vague provisions for civil penalties to be administered by the Attorney General (AG) for violations of the bill. To address this ambiguity, the author has agreed to amend the bill to establish a tiered system of enforcement.
Under the amended framework, the lowest tier of violations would apply to unknowing violations of the bill that do not create a danger of death, serious physical injury, or a catastrophic risk—for example, the late publication of a required report. These violations would be subject to a civil penalty but would not trigger immediate enforcement action. The next tier includes either knowing violations that do not result in a threat to life or safety or unknowing violations that do create a danger of death, serious physical injury, or a catastrophic risk. These violations also carry civil penalties but reflect a higher level of potential harm or culpability. The highest tier is reserved for knowing violations that do create a danger of death, serious physical injury, or a catastrophic risk. These represent the most serious offenses under the bill and would be subject to the most significant penalties. Penalties will scale according to the level of violation. For the lowest tier of offenses, the bill clarifies that developers must be provided a 30-day grace period to cure the violation after receiving notice from the AG before penalties may be imposed.
Finally, the amended bill extends enforcement provisions to include violations committed by auditors, who will also be subject to civil penalties for noncompliance.
Clarifying amendments. The author has also agreed to various clarifying, technical, and cleanup amendments. The amendments are shown below:
SECTION 1. The Legislature finds and declares all of the following:
(a) California is leading the world in artificial intelligence innovation and research through companies large and small and through the state’s remarkable public and private universities.
(b) Artificial intelligence, including new advances in foundation models, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c) The Joint California Policy Working Group on AI Frontier Models has recommended sound principles for policy in artificial intelligence.
(d) Targeted interventions to support effective artificial intelligence governance should balance the technology’s benefits and material risks.
(e) Artificial intelligence developers have already voluntarily committed to creating safety and security protocols and releasing the results of risk assessments.
(f) In building a robust and transparent evidence environment, policymakers can align incentives to simultaneously protect consumers, leverage industry expertise, and recognize leading safety practices.
(g) When industry actors conduct internal research on their technologies’ impacts, a significant information asymmetry can develop between those with privileged access to data and the broader public.
(h) Greater transparency, given current information deficits, can advance accountability, competition, and public trust.
(i) Whistleblower protections and public-facing information sharing are key instruments to increase transparency.
(j) Adverse event reporting systems enable monitoring of the post-deployment impacts of artificial intelligence.
(k) There is growing evidence that, unless they are developed with careful diligence and reasonable precaution, advanced artificial intelligence systems could pose catastrophic risks from both malicious uses and malfunctions, including artificial intelligence-enabled hacking, biological attacks, and loss of control.
(l) With the frontier of artificial intelligence rapidly evolving, there is a need for legislation to track the frontier of artificial intelligence research and alert policymakers and the public to the risks and harms from the very most advanced artificial intelligence systems, while avoiding burdening smaller companies behind the frontier.
(m) A computational threshold of 10^26 floating point operations captures the current frontier of foundation model development and captures only highly resourced developers spending hundreds of millions of dollars to develop foundation models.
(nm) In the future, foundation models developed by smaller companies or that are behind the frontier may pose significant catastrophic risk, and additional legislation may be needed at that time.
(on) It is the intent of the Legislature to create more transparency, but collective safety will depend in part on large developers taking due care in their development and deployment of foundation models proportional to the scale of the foreseeable risks.
SEC. 2. Chapter 25.1 (commencing with Section 22757.10) is added to Division 8 of the Business and Professions Code, to read:
CHAPTER 25.1. Transparency in Frontier Artificial Intelligence Act
22757.10. This chapter shall be known as the Transparency in Frontier Artificial Intelligence Act.
22757.11. For purposes of this chapter:
(a) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b) “Catastrophic risk” means a foreseeable and material risk that a large developer’s development, storage, use, or deployment of a foundation model will materially contribute to the death of, or serious injury to, more than 100 50 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or ,or loss of, property arising from a single incident, scheme, or course of conduct involving a dangerous capability.
(c) “Critical safety incident” means any of the following:
(1) Unauthorized access to, modification of, or exfiltration of, the model weights of a foundation model.
(2) Harm resulting from the materialization of a catastrophic risk.
(3) Loss of control of a foundation model causing death, bodily injury, or damage to rights in money, or loss of, property.
(4) A foundation model that uses deceptive techniques against the large developer to subvert the controls or monitoring of its large developer outside of the context of an evaluation designed to elicit this behavior.
(5) Attaining a dangerous capability or catastrophic risk threshold, as defined in the large developer’s safety and security protocol pursuant to paragraph (3) of subdivision (a) of Section 22757.12, for the first time.
(d) “Dangerous capability” means the capacity of a foundation model to do any of the following:
(1) Provide expert-level assistance in the creation or release of a chemical, biological, radiological, or nuclear weapon.
(2) Conduct or assist in a cyberattack.
(3) Engage in conduct, with limited human intervention, that would, if committed by a human, constitute the crime of murder, assault, extortion, theft, including theft by false pretense deception, or a serious or violent felony.
(4) Evade the control of its large developer or user.
(e) (1) “Deploy” means to make a foundation model available to a third party for use, modification, copying, or combination with other software.
(2) “Deploy” does not include making a foundation model available to a third party for the primary purpose of developing or evaluating the foundation model.
(f) “Large developer” means either of the following:
(1) (A) Before January 1, 2027, “large developer” means a person who has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations. meets both of the following criteria:
(i) (I) The person has trained, or initiated the training of, at least one foundation model using a quantity of computing power greater than 10^26 integer or floating-point operations.
(II) The quantity of computing power described in subparagraph (A) subclause (I) shall include computing for the original training run and any subsequent fine-tuning, reinforcement learning, or other material modifications to a preceding foundation model.
(ii) The person had annual gross revenues in excess of one hundred million dollars ($100,000,000) in the preceding calendar year.
(B) If the Attorney General does not adopt a regulation described in subparagraph (A) by January 1, 2027, the definition in paragraph (1) shall be operative until the regulation is adopted.
(g) “Foundation model” means an artificial intelligence model that is all of the following:
(1) Trained on a broad data set.
(2) Designed for generality of output.
(3) Adaptable to a wide range of distinctive tasks.
(h) “Model weight” means a numerical parameter in a foundation model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(i) “Property” means tangible or intangible property.
(ij) “Safety and security protocol” means documented technical and organizational protocols to manage, assess, and mitigate catastrophic risks.
22757.12. (a) A large developer shall write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail all of the following:
(1) How, if at all, the large developer excludes certain foundation models from being covered by its safety and security protocol because those foundation models are incapable of do not posinge material catastrophic risks and which foundation models, if any, are included.
(2) The testing procedures that the large developer uses to assess catastrophic risks from its foundation models, including risk resulting from malfunctions, misuse, and foundation models evading the control of the large developer or user.
(3)(A) Thresholds used by the large developer to identify and assess whether the foundation model has attained a dangerous capability or poses a catastrophic risk.
(B) How the large developer will assess whether those thresholds have been attained, which may include multiple tiered thresholds for different dangerous capabilities or for catastrophic risk.
(C)The actions the large developer will take if each threshold is attained.
(34) The mitigations that a large developer takes to reduce a catastrophic risk and how the large developer assesses the effectiveness of those mitigations.
(45) The degree to which the large developer’s assessments of catastrophic risk and dangerous capabilities and the effectiveness of catastrophic risk mitigations are reproducible by external entities.
(56) The extent to which, and how, a large developer will use third parties to assess catastrophic risks and dangerous capabilities and the effectiveness of mitigations of catastrophic risk.
(67) The large developer’s cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties.
(78) To the extent that the foundation model is controlled by the large developer, the procedures the large developer will use to monitor critical safety incidents and the steps that a large developer would take to respond to a critical safety incident, including, but not limited to, whether the large developer has the ability to promptly shut down copies of foundation models owned and controlled by the large developer, who the large developer will notify and the timeline on which the large developer would take these steps.
(89) The testing procedures that the large developer will use to assess and manage a catastrophic risk or dangerous capability resulting from the internal use of its foundation models, including risks resulting from a foundation model circumventing oversight mechanisms, and the schedule, specified in days, by which the large developer will report these assessments pursuant to subdivision (d).
(910) How the developer determines when its foundation models are substantially modified enough to conduct additional assessments and publish a transparency report pursuant to subdivision (c).
(b) If a large developer makes a material modification to its safety and security protocol, the large developer shall clearly and conspicuously publish the modified protocol and a justification for that modification within 30 days.
(c) Before or concurrently with deploying a new foundation model or a substantially modified version of an existing foundation model covered by the large developer’s safety and security protocol, a large developer shall clearly and conspicuously publish on its internet website a transparency report containing all of the following:
(1)(A) The results of any risk assessment or risk mitigation assessment conducted by the large developer or a third party that contracts with a large developer pursuant to its safety and security protocol, why the information gathered by the large developer or third party leads to the stated results, and the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted by the large developer pursuant to its safety and security protocol.
(2) (A) The results of any risk assessment, the steps taken to address any identified risks, and the results of any risk mitigation assessment conducted pursuant to a large developer’s safety and security protocol that is conducted by a third party that contracts with a large developer.
(B) A large developer shall disclose the time and extent of predeployment access provided to any third party described in subparagraph (A), whether or not the third party was independent, and the nature of any constraints the large developer placed on the assessment or on the third party’s ability to disclose information about its assessment to the public or to government officials.
(C) Whether a catastrophic risk threshold or dangerous capability threshold has been attained for the foundation model and any actions taken as a result.
(32) (A) If the deployment would pose a catastrophic risk, tThe reasoning behind the large developer’s decision to deploy the foundation model, the process by which the large developer arrived at that decision, and any limitations in the assessments that the large developer used to make that decision.
(B) A large developer may reuse an answer previously provided under subparagraph (A) if the rationale in question has not materially changed for the new deployment.
(d) A large developer shall clearly and conspicuously publish on its internet website any assessment of catastrophic risk or dangerous capabilities resulting from internal use of its foundation models pursuant to the schedule the developer specifies in its safety and security protocol.
(e) A large developer shall not make a materially false or misleading statement about catastrophic risk from its foundation models, or its management of catastrophic risk, or its implementation of or compliance with its safety and security protocol.
(f) (1) When a large developer publishes documents to comply with this section, the large developer may make redactions to those documents that are necessary to protect the large developer’s trade secrets, the large developer’s cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.
(2) If a large developer redacts information in a document pursuant to this subdivision, the large developer shall describe the character and justification of the redaction in any published version of the document to the extent permitted by the concerns that justify redaction and shall retain the unredacted information for five years.
22757.13. (a) The Attorney General shall establish a mechanism to be used by a large developer or a member of the public to report a critical safety incident that includes all of the following:
(1) The date of the critical safety incident.
(2) The reasons the incident qualifies as a critical safety incident.
(3) A short and plain statement describing the critical safety incident.
(b) A large developer shall report any critical safety incident pertaining to one or more of its foundation models to the Attorney General within 15 days of discovering the critical safety incident.
(c) If a large developer discovers a critical safety incident poses an imminent risk of danger of death or serious physical injury, the large developer shall disclose that incident as soon as practicable, and in no event later than 24 hours, to an authority, including any law enforcement agency or public safety agency with jurisdiction, that is appropriate based on the nature of that incident and in accordance with applicable law.
(cd) The Attorney General shall review critical safety incident reports submitted by large developers and may review reports submitted by members of the public.
(e) The Attorney General may transmit reports of critical safety incidents, reports from employees pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor’s reports pursuant to Section 22757.16 to the Legislature, Governor, the federal government or appropriate state agencies at the Attorney General’s discretion and in accordance with relevant law.
(f) A report of a critical safety incident submitted to the Attorney General pursuant to this section, reports from employees pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor’s reports pursuant to Section 22757.16 are exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(g) (1) Beginning January 1, 2027, and annually thereafter, the Attorney General shall produce a report with anonymized and aggregated information about critical safety incidents, reports from employees pursuant to Chapter 5.1 (commencing with Section 1107) of Part 3 of Division 2 of the Labor Code, and summaries of auditor’s reports pursuant to Section 22757.16 that have been reviewed by the Attorney General since the preceding report.
(2) The Attorney General shall not include information in a report pursuant to this subdivision that would compromise the trade secrets or cybersecurity of a large developer, public safety, or the national security of the United States or that would be prohibited by any federal or state law.
(3) The Attorney General shall transmit a report pursuant to this subdivision to the legislature pursuant to Section 9795 and to the Governor.
22757.14. (a) A violation of this chapter shall be subject to a civil penalty in an amount not to exceed ____ dollars ($____) per violation, unless the violation is willful or reckless, in which case the civil penalty shall be in an amount not to exceed ____ dollars ($____).
(a) A violation of this chapter by a large developer shall be subject to a civil penalty as follows:
(1) (A) For an unknowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk, a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(B) (i) In the case of a first violation subject to a civil penalty pursuant to subparagraph (A), a large developer shall be provided with a thirty (30) day right to cure the violation after notification by the Attorney General. If the violation is cured within that period, no civil penalty shall be imposed for that violation.
(ii) For the purposes of this subparagraph, a violation involving the missing or late publication or submission of a document is deemed cured when the developer publishes or submits that document, even if the publication or submission occurs after the statutory deadline.
(2) For a knowing violation that does not create a material risk of death, serious physical injury, or a catastrophic risk or an unknowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a civil penalty in an amount not to exceed one hundred thousand dollars ($100,000).
(3) For a knowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a civil penalty in an amount not to exceed one million dollars ($1,000,000) for a first such violation and in an amount not exceeding ten million dollars ($10,000,000) for any subsequent such violation.
(b) A violation of this chapter by an auditor shall be subject to a civil penalty in an amount not to exceed ten thousand dollars ($10,000).
(bc) The civil penalty shall be assessed in a civil action brought only by the Attorney General.
22757.15. (a) On or before January 1, 2027, and annually thereafter, the Attorney General may adopt regulations to update the definition of a “large developer” for the purposes of this chapter to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to well-resourced large developers at the frontier of artificial intelligence development.
(b) In developing regulations pursuant to this section, the Attorney General shall take into account all of the following:
(1) Similar thresholds used in international standards or federal law, guidance, or regulations for the management of catastrophic risk.
(2) Input from stakeholders, including academics, industry, the open-source community, and governmental entities.
(3) The extent to which a person will be able to determine, before beginning to train or deploy a foundation model, whether that person will be subject to the regulations as a large developer with an aim toward allowing earlier determinations if possible.
(4) The complexity of determining whether a person is covered, with an aim toward allowing simpler determinations if possible.
(5) The external verifiability of determining whether a person is covered, with an aim toward definitions that are verifiable by parties other than the large developer.
(c) If the Attorney General determines that less well-resourced developers, or developers significantly behind the frontier of artificial intelligence, may create substantial catastrophic risk, the Attorney General shall promptly submit a report to the Legislature, pursuant to Section 9795, with a proposal for managing this source of catastrophic risk but shall not include those developers within the definition of “large developer” without authorization in subsequently enacted legislation.
22757.16. (a) Beginning January 1st, 2030, and at least annually thereafter, a large developer shall retain an independent third-party auditor to produce a report assessing both of the following:
(1) Whether the large developer has substantially complied with its safety and security protocol and any instances of substantial noncompliance during the prior year.
(2) Any instances where the large developer's safety and security protocol has not been stated clearly enough to determine whether the large developer has complied.
(b) A large developer shall allow the third-party auditor access to all materials produced to comply with this Act and any other materials reasonably necessary to perform the assessment required under subsection (a).
(c) A large developer shall retain the auditor’s report for 5 years.
(d) In conducting the audit, the auditor shall employ or contract one or more individuals with expertise in corporate compliance and one or more individuals with technical expertise in the safety of foundation models.
(e) Within 30 days after completing an audit, the auditor shall transmit to the Attorney General a high-level summary of the auditor’s report.
(f) The high-level summary required by subdivision (e) shall fairly present, in all material respects, the outcome of the audit, and an auditor shall not knowingly include a material misrepresentation or omit a material fact in either that summary or the auditor’s report.
SEC. 3. Section 11546.8 is added to the Government Code, to read:
11546.8. (a) There is hereby established within the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as “CalCompute.”
(b) The consortium shall develop a framework for the creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1) Fostering research and innovation that benefits the public.
(2) Enabling equitable innovation by expanding access to computational resources.
(c) The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d) CalCompute shall include, but not be limited to, all of the following:
(1) A fully owned and hosted cloud platform.
(2) Necessary human expertise to operate and maintain the platform.
(3) Necessary human expertise to support, train, and facilitate the use of CalCompute.
(e) The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f) (1) On or before January 1, 2027, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for the creation and operation of CalCompute.
(2) The report required by this subdivision shall include all of the following elements:
(A) A landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure.
(B) An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
(C) Recommendations for the governance structure and ongoing operation of CalCompute.
(D) Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E) An analysis of the state’s technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute
(F) A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G) Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g) The consortium shall, consistent with state constitutional law, consist of 14 members as follows:
(1) Four representatives of the University of California and other public and private academic research institutions and national laboratories appointed by the Secretary of Government Operations.
(2) Three representatives of impacted workforce labor organizations appointed by the Speaker of the Assembly.
(3) Three representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates appointed by the Senate Rules Committee.
(4) Four experts in technology and artificial intelligence to provide technical assistance appointed by the Secretary of Government Operations.
(h) The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
(i) The consortium shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
(j) If CalCompute is established within the University of California, the University of California may receive private donations for the purposes of implementing CalCompute.
(k) This section shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
SEC. 4. Chapter 5.1 (commencing with Section 1107) is added to Part 3 of Division 2 of the Labor Code, to read:
CHAPTER 5.1. Whistleblower Protections: Catastrophic Risks in AI Foundation Models
1107. For purposes of this chapter:
(a) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b) “Catastrophic risk” has the meaning defined in Section 22757.11 of the Business and Professions Code.
(c) “Large developer” has the meaning defined in Section 22757.11 of the Business and Professions Code.
(d) “Employee” means a person who performs services for an employer, including both of the following:
(1) A contractor, subcontractor, or an unpaid advisor involved with assessing, managing, or addressing catastrophic risk, including all of the following:
(A) An independent contractor.
(B) A freelance worker.
(C) A person employed by a labor contractor.
(D) A board member.
(2) Corporate officers.
(e) “Foundation model” has the same meaning as defined in Section 22757.11 of the Business and Professions Code.
(f) “Labor contractor” means an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employer’s usual course of business.
1107.1. (a) A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
(1) The large developer’s activities pose a catastrophic risk.
(2) The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(b) A large developer shall not enter into a contract that prevents an employee from making a disclosure protected under Section 1102.5.
(c) A large developer shall not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the large developer related to the assessment, management, or addressing of catastrophic risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following
(1) The large developer’s activities pose a catastrophic risk.
(2) The large developer has violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code.
(d) An employee may use the hotline described in Section 1102.7 to make reports described in subdivision (a).
(e) A large developer shall provide a clear notice to all employees of their rights and responsibilities under this section, including by doing either of the following:
(1) At all times posting and displaying within any workplace maintained by the large developer a notice to all employees of their rights under this section, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice.
(2) At least once each year, providing written notice to each employee of the employee’s rights under this section and ensuring that the notice is received and acknowledged by all of those employees.
(f) (1) A large developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the large developer if the employee believes in good faith that the information indicates that the large developer’s activities present a catastrophic risk or that the large developer violated Chapter 25.1 (commencing with Section 22757.10) of Division 8 of the Business and Professions Code, including a monthly update to the person who made the disclosure regarding the status of the large developer’s investigation of the disclosure and the actions taken by the large developer in response to the disclosure.
(2) (A) Except as provided in subparagraph (B), the disclosures and responses of the process required by this subdivision shall be shared with officers and directors of the large developer at least once each quarter.
(B) If an employee has alleged wrongdoing by an officer or director of the large developer in a disclosure or response, subparagraph (A) shall not apply with respect to that officer or director.
(g) The court is authorized to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
(h) In a civil action brought pursuant to this section, once it has been demonstrated by a preponderance of the evidence that an activity proscribed by this section was a contributing factor in the alleged prohibited action against the employee, the large developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
(i) (1) In a civil action or administrative proceeding brought pursuant to this section, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
(2) Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
(3) In addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper.
(4) Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
(5) An order authorizing temporary injunctive relief shall remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review pursuant to subdivision (b) of Section 98.74, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a large developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
(j) Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant to this section shall not be stayed pending appeal.
(k) (1) This section does not impair or limit the applicability of Section 1102.5.
(2) The remedies provided by this section are cumulative to each other and the remedies or penalties available under all other laws of this state.
SEC. 5. (a) The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
(b) This act shall be liberally construed to effectuate its purposes.
(c) The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
(d) This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a large developer.
(e) This act shall not apply to the extent that it is preempted by federal law.
SEC. 6. The Legislature finds and declares that Section 2 of this act, which adds Chapter 25.1 (commencing with Section 22757.10) to Division 8 of the Business and Professions Code, imposes a limitation on the public’s right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in critical safety incident reports, summaries of auditor’s reports, and reports from employees may contain information that could threaten public safety or compromise the response to an incident if disclosed to the public.
ARGUMENTS IN SUPPORT: Secure AI Project, co-sponsors of the bill, alongside a coation of technology equity advocacy groups write in support:
The California Report on Frontier AI Policy, while it does not endorse any specific legislation, forms the foundation for SB 53. Established by Governor Newsom in 2024 and led by Dr. Fei-Fei Li, Dr. Jennifer Tour Chayes and Mariano-Florentino Cuéllar, the report is anchored on the notion of “trust but verify” and calls for more transparency into the safety practices of AI companies, adverse event reporting requirements, and whistleblower protections. SB 53 implements these principles.
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The Report stated that “some risks have unclear but growing evidence...AI-enabled hacking or biological attacks, and loss of control” – the risks that SB 53 aims to address and gather more evidence about. Advanced AI is currently mostly unregulated, and these risks are currently being managed by companies themselves without any requirement that they inform the public about their risk management practices or report serious incidents. SB 53 addresses this much needed gap by implementing four key recommendations from the report.
First, the Report argued that “transparency into the risks associated with foundation models, what mitigations are implemented to address risks, and how the two interrelate is the foundation for understanding how model developers manage risk.” SB 53 implements this recommendation as a requirement for large AI developers to write, publish, and follow safety and security protocols to manage the most severe risks. This is in line with voluntary commitments that companies have already made. Rather than prescribe specific technical standards that companies must take, the bill simply requires companies to be transparent about the approaches they are using. Some of the specific required elements of safety protocols, such as a requirement to manage risks related to internal use of AI models and cybersecurity policies, directly mirror recommendations in the Report. Others mirror components of the Stanford Foundation Model Transparency Index, which is cited prominently in the Report.
Second, the Report stated that “transparency into pre-deployment assessments of capabilities and risks, spanning both developer-conducted and externally conducted evaluations, is vital given that these evaluations are early indicators of how models may affect society and may be interpreted (potentially undesirably) as safety assurances.” SB 53 accomplishes this with a requirement that large developers publish transparency reports that include the results of their pre-deployment assessments of catastrophic risk. The Report also argues that “transparency into the safety cases used to assess risk provides clarity into how developers justify decisions around model safety,” which forms the basis for 22757.12(c)(3).
Third, the Report concluded that “an adverse event reporting system that combines mandatory developer reporting with voluntary user reporting maximally grows the evidence base.” SB 53 takes exactly this approach by establishing a tightly defined set of critical safety incidents that AI developers are required to report to the Attorney General. It would also allow members of the public to optionally submit reports.
Finally, the Report recommends strengthening whistleblower protections, pointing out that “actions that may clearly pose a risk and violate company policies...may not violate any existing laws. Therefore, policymakers may consider protections that cover a broader range of activities, which may draw upon notions of ‘good faith’ reporting on risks found in other domains such as cybersecurity.” This recommendation is mirrored in SB 53, which allows employees to report evidence of catastrophic risks as well as violations of SB 53 itself to government authorities with legal protections against retaliation.
SB 53 only applies to the largest AI developers – those training models with more than 10^26 floating point operations (FLOPs). These are companies spending hundreds of millions or billions of dollars to train the most advanced AI models. It would impose no burden on smaller companies and the requirements it imposes on large companies are minimal compared to what companies are already voluntarily doing. The Report argues that “policymakers should ensure that mechanisms are in place to adapt thresholds over time—not only by updating specific threshold values but also by revising or replacing metrics if needed.” It also suggests specific criteria that thresholds should be evaluated for. Following this recommendation, SB 53 allows the Attorney General to update the definition of “large developer” through regulation while considering the same factors described in the report. Regardless of any update, the Attorney General must only include “well-resourced large developers at the frontier of artificial intelligence development” in the scoping of the bill. If legislation is needed to cover other developers, the Attorney General is instructed to write a report to the Legislature requesting it.
Finally, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California’s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. This mirrors a similar computing cluster that is already being established in New York state. We support this groundbreaking effort, which would advance and democratize AI research in California.
SB 53 thoughtfully implements the recommendations of the Report by combining a low-burden transparency and reporting regime with a public compute cluster that will broaden access for AI researchers and startups in California. This is a commonsense approach that will strengthen the AI ecosystem, benefiting both companies and the public interest.
For all these reasons, we respectfully urge your support of this important measure.
ARGUMENTS IN OPPOSITION: In opposition to the bill, Chamber of Progress argues:
On behalf of the Chamber of Progress, a tech industry association supporting public policies to build a more inclusive society in which all people benefit from technological advances, we respectfully urge you to oppose SB 53, based on its recent amendments.
The definition of “catastrophic risk” remains vague and overreaching
While the amended bill replaces the term “critical risk” with “catastrophic risk,” the underlying problem persists. The definition remains overly expansive and ambiguous, capturing a wide array of hypothetical scenarios that may not reflect real-world AI capabilities or threats.
Under Section 22757.11(b), the definition of “catastrophic risk” includes scenarios where a foundation model is “materially likely” to cause harm, potentially due to misuse or malicious inputs. However, this standard is vague, lacks clear and objective thresholds, and leaves room for subjective interpretation by whistleblowers or regulators. In a rapidly evolving field like AI, such ambiguity could unfairly penalize developers who are acting responsibly.
In addition, the inclusion of highly abstract risks, such as the evasion of human control under Section 22757.12(a)(2), creates significant uncertainty. Without clear technical criteria, companies may face liability or investigation based on assumptions about what a model might enable rather than what it has demonstrably done. This uncertainty undermines research and commercial deployment in California and could push critical AI development efforts out of state or abroad.
The $100,000,000 compute cost threshold risks misidentifying frontier AI models
SB 53’s use of an arbitrary $100,000,000 compute cost threshold to determine eligibility for protections is an inherently flawed method for identifying frontier AI models. This threshold may result in the overinclusion of developers working on benign systems while potentially excluding smaller models that pose significant real-world risks. It also ignores the constantly changing cost of compute.
A more effective approach would involve a threshold based on model capabilities, deployment context, and specific use cases rather than relying solely on computational costs.
SB 53’s extensive safety and security protocols create impractical burdens for AI developers
SB 53 imposes comprehensive safety and security requirements on AI developers, as outlined in Section 22757.12(a), including risk testing, deployment practices, and escalation procedures. While these objectives are important, the bill demands an impractical level of detailed planning and documentation for every conceivable misuse scenario, many of which are speculative or unrealistic.
This exhaustive approach compels developers to allocate significant time and resources toward preparing for hypothetical risks rather than addressing actual, demonstrable harms. For startups and smaller companies, these extensive protocols create a heavy administrative burden that diverts critical resources away from innovation and the timely deployment of beneficial AI technologies.
Additionally, Section 22757.12(c)’s requirement that developers publish detailed transparency reports, before or at the time of deploying a new or substantially modified foundation model, creates significant risks to both competitiveness and operational security.
Although redactions are permitted under subsection (f), the requirement to publish the “character and justification” of redacted material could still inadvertently expose business-sensitive strategies or vulnerabilities. This level of forced transparency goes beyond reasonable accountability and may discourage responsible companies from operating in California. It also creates opportunities for misuse by malicious actors who could exploit disclosed model weaknesses or mitigation gaps.
In fast-moving AI markets, publication of this level of detail erodes a developer’s ability to maintain a competitive edge and deters innovation by raising legal and reputational risks associated with even speculative harms.
Taking an opposed unless amended position, the California Chamber of Commerce in a coalition with other technology trade organizations argue:
As a general matter, we see areas where the bill diverges from the final findings of Governor Newsom’s Joint California Policy Working Group on Frontier Models, which arose out of his veto of SB 1047 (Wiener, 2024) and have drawn your attention to certain areas where alignment can be better sought. While we cannot support SB 53 in its current form, we anticipate that amendments can be made to address these issues. While these are only our initial views given the complexity of the issues raised by the amendments, we hope there is the opportunity to work on this legislation with you to get this important policy right. In the interim, we believe this provides a strong overview of a path forward from our perspective to ensure SB 53 will meaningfully advance AI safety; rather than risk imposing substantial compliance costs and technical and legal challenges for certain AI developers to assess and prevent any and all downstream risks once their models are available for others to modify and deploy.
SB 53 should ideally focus on model risk, not developer size—otherwise, it will likely fail to address concerns about smaller, specialized models being a source of catastrophic risk
We are concerned about the scope of the bill and its focus on “large developers” to the exclusion of other developers of models with advanced capabilities that pose risks of catastrophic harm. In his veto of SB 1047 last year, Governor Newsom criticized this approach stating:
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
As amended July 8th, SB 53 is neither sufficiently focused on, nor does it provide any definition of what constitutes a “frontier” model – effectively equating today’s frontier AI models with foundation models and defining the threshold for covered models from there. Specifically, under the premise that today’s frontier of foundation model development reflects a computational threshold of 10^26 floating point operations (or “FLOPs”), which in turn “captures only highly resourced developers spending hundreds of millions of dollars to develop foundation models” the bill defines covered models based on the developer’s resources and then inappropriately places the burden on developers to justify why models that are above these thresholds should not be included in safety protocols.
While the Governor’s report discusses foundational models, it is clear that the focus should be on foundation models with such advanced capabilities that they pose novel and potentially severe risks. We urge SB 53 to be more focused on risk so as to not capture all big foundational models.
Consistent with our position in SB 1047, equating model size to risk manages to make the bill simultaneously overly broad and too narrow as smaller and/or less performant models can present much greater risks than large/higher performant ones. Many small entities can develop hugely influential and potentially risky models with similar capabilities to the models developed by “large developers”, as demonstrated by the Chinese company DeepSeek. As noted above, upon vetoing SB 1047, the Governor commissioned experts in the field to form a Working Group on Frontier AI, which has since validated such concerns in their Final Reports, finding that small companies may create powerful models that pose safety risks. Yet inexplicably, such models get excluded here. As a result, the bill both fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones.
There are two concerns that relying exclusively on FLOPs raises. First, it is a flawed proxy for catastrophic risks that ignores a model’s capabilities. Second, it risks becoming obsolete in a short period of time requiring the law to change yet again. The Final Report also noted the variety of reasons why FLOPs are an unreliable proxy for risk and recommended using them just for screening models if used at all, saying “we conclude that if training compute thresholds are used at all, they may function best when used as an initial filter to cheaply screen for entities that may warrant greater scrutiny.” The report was also critical of other proxies related to size, such as money spent on computing power for model development – a secondary threshold attempted in SB 1047 which we would caution against as a fallback here.
Adaptability is important—as is the Legislature’s authority over major policy decisions
The Governor’s veto of SB 1047 emphasized the importance of adaptability—a theme echoed in the Governor’s Working Group Final Report, which states that “[m]ost fundamentally, the pace of technological change will require adaptive approaches that allow for thresholds to be flexible and readily updated to avoid ossification.” We appreciate that SB 53, as currently drafted, attempts to incorporate a mechanism for adaptability by allowing the scope of the law to evolve as science progresses. Once again, however, its success is limited by tethering that adaptability to developer size rather than model risk, and by limiting the role of scientific evidence to which the thresholds can adapt.
Specifically, the bill authorizes the AG to annually revise the definition of a “large developer” to ensure it captures “well-resourced large developers at the frontier of artificial intelligence development.” (See proposed Sec. 22757.15(a).) As noted above, this approach is based on a presumption that size or resources are a sufficient proxy for risk or capability, and in doing so, it disregards the possibility that a risk- or capability-based framework may be more appropriate for determining scope.
Comparatively, proposed Section 22757.15(c) establishes a different mechanism for addressing risk from smaller or less well-resourced developers. If the AG determines that such developers, or those significantly behind the frontier of AI, may nonetheless pose a substantial catastrophic risk, the AG must submit a detailed report to the Legislature along with a proposal for addressing that risk. Yet the AG is expressly prohibited from unilaterally expanding the “large developer” definition to include those entities without legislative approval.
This distinction reflects a tension at the heart of the bill. On one hand, in legislation aimed at promoting safety, it is essential to have mechanisms that can respond quickly to rapid advancements in model capabilities, particularly those nearing thresholds associated with catastrophic risk. On the other hand, it is equally important to avoid overreach – whether by stifling innovation in lower-resourced environments or by granting non-legislative bodies (particularly ones without subject matter expertise) the authority to redefine the law’s scope without adequate public input or legislative oversight. By no means should the AG be given unfettered or unchecked authority to make decisions as to who is and is not subject to this law. Regardless, a genuinely adaptive or iterative regulatory framework responds to all relevant scientific evidence, not some of it.
Furthermore, we note that granting the AG authorization to annually pass regulations does not have a different impact than merely granting the AG authorization to pass regulations. The AG could just as easily decide to examine these issues and pass regulations every five years. There is nothing requiring the AG to examine the evidence annually. Arguably, an annual review each interim by a legislative oversight committee would provide greater assurance of an adaptive definition than such an authorization, and it would do so without giving the AG such unfettered authority and without giving away legislative authority. That is not to say that the committee oversight process would be the appropriate alternative given all factors to be considered; simply that SB 53 has not yet found the right approach. Anticipating the development of national or international benchmarks for evaluating model capabilities in the text could be an alternative approach supported by the Working Group’s final report.
The definitions of catastrophic risk, critical safety incidents, and dangerous capabilities are vague and contradictory
SB 53 relies on three terms that appear to include unique characterizations of risks for which developers must evaluate and mitigate against. Unfortunately, the terms cover similar grounds, making use of these three separate yet similar terms is both confusing and overly complex. For example, catastrophic risk is defined via thresholds for injuries or death to persons and dollars of loss resulting from dangerous capabilities. This creates concern given the inclusion of widely available capabilities from existing software tools, such as the ability to “assist in a cyberattack” or commit a crime with “limited human intervention” within the definition of “dangerous capabilities”. While dangerous capabilities are only relevant to the extent they create catastrophic risk, as described above, “catastrophic risk” doesn’t focus on risk of harm posed by unique risks associated with advanced AI capabilities.
It appears to include currently existing risks that AI models of all sizes may be misused in ways that can cause multiple deaths and high property damage, much like the internet and computers can be used to disrupt critical infrastructures with ransomware attacks that create similar risks of bodily injury, death, and financial losses and damage.
Next, there is also an overbreadth issue, starting with the proposed definition of “catastrophic risk” which requires that the harm will materially contribute in death or serious injury to more than 100 people or more than one billion dollars in damage to rights in money or property arising from a single incident, scheme, or course of conduct involving a dangerous capability. The definition is contradictory, listing both single incident, as well as scheme or course of conduct, vastly expanding the scope of the bill. It is unnecessarily and detrimentally overly broad because a focused SSP is more effective and would ideally be focused on certain categories of risks that exclude things such as blackmail, theft by deception, federation operation, publicly available data, nonmaterial contribution, or cyber threats for example.
Finally, as currently drafted, SB 53’s overall “critical safety incidents” approach is fundamentally incompatible with the way the AI ecosystem, for both open-source and closed models, as it would impose reporting obligations regarding downstream developer incidents that are infeasible for a developer who does not control the AI system involved in the incident. We note that whereas the Governor’s Work Group report recognized the full AI ecosystem value chain, SB 53 still needs to more fully recognize the roles of not just the original developer of a foundational model but also of those unaffiliated third parties who may modify and/or build on top of a foundation model. The bill should clarify these provisions to reflect the realities of the ecosystem, including open-source models, eliminate vague triggers, and provide flexibility in reporting timelines to accommodate investigation.
Ultimately, each of the definitions need reworking to focus on advanced capabilities of frontier models and the unique risks they may pose. While doing so, we suggest that an effort be made to unify the terminology and substantive criteria as much as possible to align to the focus recommended in the Final Report. Most important, we feel it critical that SB 53 better align with the marginal risk standard adopted in the report. As framed by the Governor’s Work Group: “that policymakers center their calculus around the marginal risk: Do foundation models present risks that go beyond previous levels of risks that society is accustomed to from prior technologies, such as risks from search engines?”
Required disclosures are far too specific, which poses significant security risks and exploitation by bad actors
As currently drafted, proposed Section 22757.12(a) requires that a large developer “write, implement, and clearly and conspicuously publish on its internet website a safety and security protocol that describes in specific detail” specified information detailing the testing procedures that the large developer uses to assess catastrophic risks from its foundation model, among other things.
We have significant concerns about describing many of the required elements in “detail.” First, “describing in detail” certain items in a “protocol,” such as “testing procedures,” “mitigations,” and whether evaluations are reproducible are elements of a protocol or development framework that should be adaptable to future scientific developments and flexible based on the capabilities displayed of a given model. A developer should not become beholden to use specific mitigation techniques because they are in their published protocol if a better more effective mitigation is available. Whether capabilities associated with catastrophic risks are reproducible may depend on the model, and the risk was detected.
But also, required disclosures should allow for a level of generality to protect models from exploitation and attack. For example, the requirement in proposed subdivision (a)(6) that a developer describe in detail the “cybersecurity practices and how the large developer secures unreleased model weights from unauthorized modification or transfer by internal or external parties,” creates security vulnerabilities for the developer by unnecessarily disclosing detailed information about their cyber protections aiding would be attackers. This is almost akin to a bank putting a blueprint of the location of its security cameras and the protocols of their security guards, and the frequency with which the vault is emptied, online. Once again, as noted at the outset of this letter, the Final Report advised caution for such reasons, stating:
General details about risks of foundation models can be made public without undermining security, especially if these risks have been demonstrated in other foundation models or AI technologies. Specific details about concrete vulnerabilities should be disclosed carefully, with advanced notice to actors in the supply chain who are able to remediate them prior to broader disclosure.
Whistleblower provisions
The whistleblower protection chapter needs additional protection against abuse, which could have an impact on a developer’s trade secrets, intellectual property, and reputation. Limiting this protection to those involved in critical risk as well as corporate officers helps to mitigate abuse, but would still capture a potentially large group of people given that are “involved with assessing, managing, or addressing critical risk” which could include anyone who works on the trust and safety and testing of an AI model, not just those working at the highest level of assessment of critical risk, and given the size of the companies in scope, there’s a potential for abuse. There are different mechanisms to mitigate this risk which we are exploring.
Also, SB 53 expands “employee” to include contractors and independent workers. The definition should not be changed to include non-employees like contractors. This creates unnecessary risks of misclassification later. The bill could achieve the same stated end to create a definition for the bill of a “covered person” but that includes both employees and contractors as covered individuals.
Additional considerations, including areas of competing viewpoints in industry: something the Legislature must take seriously to avoid granting competitive advantages
There are other considerations as a result of the July 8th amendments that include the following:
- SSP/protocol requirements lack an intent standard for materially false or misleading statements (see proposed Section 22757.12(e)): the prohibition on false and misleading statements needs an intent standard such as “intentionally”, or at least “intentionally or recklessly”. Note: whereas knowingly infers an awareness of one’s action, intentionally refers to a conscious objective to cause a result (i.e., something done with reason and purpose).
- Incident reporting specificity and timeframe: proposed Section 22757.13(b) requires reporting within 15 days but does not provide flexibility for investigation timeline. Even if 15 days is a reasonable reporting period, requirements should be flexible because all facts may not be known within 15 days of discovery.
- Upstream versus downstream developers: we greatly appreciate the effort to focus on transparency and liability in SB 53, compared to SB 1047. However, it still needs to better delineate responsibilities between upstream and downstream developers. Developers who pretrain models should not be held liable for fine-tuning or modifications by downstream developers. Clarification is needed to ensure the initial developer is not treated as the developer of a subsequently modified model.
- Transparency report: the requirement for a new “transparency report” for every new foundation model should be eliminated. Instead, the bill should allow developers to rely on existing transparency practices such as model cards, requiring only a high-level summary of evaluations and the involvement of third-party assessments, if any.
- Confidentiality: we hope to see redactions be broadened beyond trade secrets and cybersecurity information, for example to cover other confidential or proprietary information.
Finally, with respect to enforcement, we ask that the bill grant businesses at least a 60 day right to cure, to ensure that law focuses on compliance and not punishment. In addition, given the highly detailed requirements of the bill as drafted, we think enforcement efforts should be focused on material failures to comply rather than also covering technical paperwork errors.
Again, while we understand your focus on the issue of AI models and hope to continue working with you on the issue, the correct approach for California, given the immense promise of this technology, would be to focus on actual risks and harms, and to provide appropriate and secure disclosures. As drafted, however, for all the aforementioned reasons, we must OPPOSE UNLESS AMENDED SB 53 (Wiener).
REGISTERED SUPPORT / OPPOSITION:
Support
Ai for Animals
Ai Futures Project
Ai Lab Watch
Ai Policy Tracker
All Girls Allowed
Apart Research
Association for Long Term Existence and Resilience (ALTER)
Berkeley Existential Risk Initiative (BERI)
Center for Ai and Digital Policy
Center for Ai Policy
Center for Human-compatible Ai
Center for Youth and Ai
Children's Advocacy Institute, University of San Diego School of Law
Common Sense Media
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Earningsstream LLC
Economic Security California Action
Elicit
Encode
Encode Ai Corporation
Encode Justice
Eon Systems
Existential Risk Observatory
Frontlines Foundation
Indivisible California Statestrong
Momentum
Nonlinear
Oakland Privacy
Redwood Research
Secure Ai Future
Secure Ai Project
Seiu Califonia
Tech Oversight California
Techequity Action
The Brandes Lab At Nyu
The Midas Project
The Signals Network
Trevi Digital Assets Fund
University of California
Oppose
Chamber of Progress
Silicon Valley Leadership Group
Oppose Unless Amended
California Chamber of Commerce
Computer and Communications Industry Association
Insights Association
Technet
Analysis Prepared by: John Bennett / P. & C.P. / (916) 319-2200
For a discussion of issues with defining frontier models, see “California Report on Frontier AI Policy” (June 17, 2025), pp. 36-40, https://www.cafrontieraigov.org/. ↑
Pablo Arredondo, “GPT-4 Passes the Bar Exam: What That Means for Artificial Intelligence Tools in the Legal Profession” (Apr. 19, 2023), https://law.stanford.edu/2023/04/19/gpt-4-passes-the-bar-exam-what-that-means-for-artificial-intelligence-tools-in-the-legal-industry/. ↑
“International AI Safety Report,” AI Action Summit (Jan. 2025), p. 38, https://assets.publishing.service.gov.uk/media/679a0c48a77d250007d313ee/International_AI_Safety_Report_2025_accessible_f.pdf. ↑
Id. at p. 44. ↑
Id. at pp. 16-17. ↑
Id. at p. 27 ↑
Introducing OpenAI o3 and o4-mini OpenAI (Apr. 16, 2025), https://openai.com/index/introducing-o3-and-o4-mini/. ↑
Sam Altman, The Intelligence Age (Sept. 23, 2024), https://ia.samaltman.com/. ↑
Kyungtae Kim, “What is AGI, and when will it arrive?: Big Tech CEO Predictions” (Mar. 20, 2025), https://www.giz.ai/what-is-agi-and-when-will-it-arrive/; see also Kokotajlo et al, “AI 2027,” (Apr. 3, 2025), https://ai-2027.com/. ↑
John Markoff, “The Coming Superbrain,” New York Times (May 23, 2009), www.nytimes.com/2009/05/24/weekinreview/24markoff.html. ↑
Cade Metz, “Why We’re Unlikely to Get Artificial General Intelligence Anytime Soon,” New York Times (May 16, 2025), https://www.nytimes.com/2025/05/16/technology/what-is-agi.html. ↑
International AI Safety Report, supra, at pp. 17-21. The report does not address Lethal Autonomous Weapon Systems, which are typically narrow AI systems specifically developed for that purpose. (See id. at pp. 26-27.) ↑
Id. at p. 177 ↑
Cat Zakrzewski and Pranshu Verma, “New Hampshire opens criminal probe into AI calls impersonating Biden,” Washington Post, February 6, 2024, www.washingtonpost.com/technology/2024/02/06/nh-robocalls-ai-biden/. ↑
Emma Woollacott, “Yes, The Bots Really Are Taking Over The Internet”, Forbes (Apr. 16, 2024). https://www.forbes.com/sites/emmawoollacott/2024/04/16/yes-the-bots-really-are-taking-over-the-internet/. ↑
F. Salvi, M. H. Ribeiro, R. Gallotti, R. West, “On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial,” arXiv [cs.CY] (2024); http://arxiv.org/abs/2403.14380. ↑
Sharma et al, “Towards Understanding Sycophancy in Language Models” Arxiv (2023), https://arxiv.org/abs/2310.13548. ↑
Meghan Bobrowsky, “Zuckerberg’s Grand Vision: Most of Your Friends Will Be AI,” Wall Street Journal (May 7, 2025), https://www.wsj.com/tech/ai/mark-zuckerberg-ai-digital-future-0bb04de7?msockid=396cc204796e68e336e7d64978db69ac. ↑
Angela Yang, “Researchers secretly infiltrated a popular Reddit forum with AI bots, causing outrage,” NBC News (Apr. 29, 2025), https://www.nbcnews.com/tech/tech-news/reddiit-researchers-ai-bots-rcna203597. ↑
Introduction to AI Safety, Ethics, and Society, supra, at p. 11. ↑
International AI Safety Report, supra, at p. 72. ↑
Soice et al, “Can large language models democratize access to dual-use biotechnology?”
https://arxiv.org/pdf/2306.03809. To mitigate these risks, the authors propose measures such as third-party evaluations of LLMs before their release, curating training datasets to exclude harmful content, and implementing stringent screening of DNA synthesis orders. ↑
Fabio Urbina et al. “Dual use of artificial-intelligence-powered drug discovery”. In: Nature Machine Intelligence 4 (2022), pp. 189–191. ↑
Mummert et al., “What is reinforcement learning?” IBM Developer (September 15, 2022), https://developer.ibm.com/learningpaths/get-started-automated-ai-for-decision-making-api/what-is-automated-ai-for-decision-making./. ↑
Brian Christian, The Alignment Problem: Machine Learning and Human Values (Norton 2020, 1st ed.), pp. 9-11. ↑
International AI Safety Report, supra, at pp. 100-107. ↑
CAPTCHA is an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart.” ↑
OpenAI, “GPT-4 System Card,” https://cdn.openai.com/papers/gpt-4-system-card.pdf. ↑
Meinke et al, “Frontier Models are Capable of In-Context Scheming,” arXiv (Jan. 2025), https://arxiv.org/pdf/2412.04984. ↑
Harry Booth, “When AI Thinks It Will Lose, It Sometimes Cheat, Study Finds,” Time (Feb. 19, 2025), https://time.com/7259395/ai-chess-cheating-palisade-research/. ↑
System Card: Claude Opus 4 & Claude Sonnet 4, pp. 27, https://www.anthropic.com/claude-4-system-card. ↑
Center for AI Safety, “Statement on AI Risk: AI Experts and Public Figures Express Their Concern about AI Risk” (2024), https://www.safe.ai/work/statement-on-ai-risk. ↑
Id. at pp. 123-126. ↑
Dan Hendryks, Introduction to AI Safety, Ethics, and Society, p. 12, https://drive.google.com/file/d/1uph559W-ASR4MEn6M_7Mb3lqQTapC_gZ/view?pli=1. ↑
Bommasani,and Singer et al.. “The California Report on Frontier AI Policy.” The Joint California Policy Working Group on AI Frontier Models. June 17,2025. p 39. ↑
Id. at p. 26. ↑
Id. at p. 19. ↑
Id. at p. 21. ↑
Id.at p. 35. ↑
Id.at p. 29. ↑
RAISE Act can be found at https://www.nysenate.gov/legislation/bills/2025/A6453/amendment/A. ↑
Bommasani, supra. p.12. ↑
Id. at. p. 12. ↑
Id. at. p. 12. ↑
“The need for transparency in Frontier Ai”, Anthropic (July 7, 2025), https://www.anthropic.com/news/the-need-for-transparency-in-frontier-ai. ↑
Bommasani, supra. p. 31. ↑
Id. at p. 15. ↑
Id. at p. 27-28. ↑
Id. at p. 42. ↑
Date of Hearing: July 1, 2025
ASSEMBLY COMMITTEE ON JUDICIARY
Ash Kalra, Chair
SBPCA Bill Id:SB 53
Author:(Wiener) – As Amended Ver:May 23, 2025
SENATE VOTE: 37-0
SUBJECT: CalCompute: foundation models: whistleblowers
KEY ISSUES:
- Should California establish a consortium at the Government Operations Agency to make recommendations for the development of “CalCompute,” a public cloud computing cluster?
- Should a developer of artificial intelligence be prohibited from making or enforcing any rule or contract provision that prohibits an employee or contractor from disclosing information about any developer activities that pose a “critical risk,” as defined?
SYNOPSIS
This bill is part of a broader effort by the Legislature to confront the growing use of artificial intelligence (AI) in a way that encourages innovation, while at the same time guarding against foreseeable risks. SB 53 addresses these twin aims in two ways. First, the bill would establish a consortium within the Government Operations Agency. It requires the consortium to develop a framework for a public cloud computing cluster, to be known as “CalCompute” to advance the development and deployment of AI that is safe, ethical, equitable, and sustainable.
The second part of this bill – and the one most relevant to this Committee – seeks to provide whistleblower protections to employees of AI developers. Specifically, the bill would prohibit an AI developer from making or enforcing any rule or contract provision that prohibits an employee or contractor from disclosing information about any developer activity that creates a “critical risk.” The bill also prohibits the developer from retaliating against an employee who makes such a disclosure. The bill defines “critical risk” to mean any foreseeable and material risk that will result in the death of, or serious injury to, more than 100 people, or more than $1 billion in monetary or property damage, but only if this damage comes about through enumerated means. As discussed in the analysis, the definition of “critical risk” requires such serious harm, and only by enumerated means, that it may not provide much, if any, whistleblower protection. The analysis suggests two possible amendments that the author may consider as the bill moves forward should it be approved by this Committee.
The bill is sponsored and supported by a coalition of advocates for safe and responsible AI development. If approved, it will be referred to the Assembly Privacy Committee. The author intends to take other amendments in that Committee on matters that mostly fall within that Committee’s jurisdiction; because those forthcoming amendments are not in the bill before the Committee, they are not addressed in this analysis.
SUMMARY: Promotes responsible AI development by establishing a consortium to develop “CalCompute,” a public cloud computing platform, and provides whistleblower protection to employees and contractors of AI developers who disclose “critical risks” posed by the development or deployment of an AI foundational model. Specifically, this bill:
- Establishes within the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable. Requires the consortium to make reasonable efforts to establish CalCompute within the University of California.
- Specifies that the consortium shall consist of fourteen members and include representatives from the University of California, impacted workforces, and other stakeholder groups and experts with experience in technology and artificial intelligence. Provides that eight members shall be selected by the Secretary of the Government Operations Agency. The President Pro Tempore of the Senate and the Speaker of the Assembly shall each appoint three members.
- Prohibits a developer from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliating against an employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the employee, or another employee who has the authority to investigate the issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The developer’s activities pose a critical risk.
- The developer has made false or misleading statements about its management of critical risk.
- Prohibits a developer from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an organization or entity that provides specified services to the developer from disclosing information to the Attorney General, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloses either of the following:
- The developer’s activities pose a critical risk.
- The developer has made false or misleading statements about its management of critical risk.
- Defines “critical risk” for purposes of the above to mean a foreseeable and material risk that a developer’s development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than one billion dollars in damage to rights in money or property, through any of the following:
- The creation and release of a chemical, biological, radiological, or nuclear weapon.
- A cyberattack.
- A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
- A foundation model evading the control of its developer or user.
EXISTING LAW:
- Establishes the Department of Technology within the Government Operations Agency and requires the department to conduct a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by any state agency. (Government Code Sections 11545-11546.)
- Prohibits employers and any person acting on behalf of the employer from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified. Employers and their agents are also prohibited from retaliating against an employee for such conduct. (Labor Code Section 1102.5.)
- Requires the office of the Attorney General to maintain a whistleblower hotline to receive calls from persons who have information regarding possible violations of state or federal statutes, rules, or regulations, or violations of fiduciary responsibility by a corporation or limited liability company to its shareholders, investors, or employees. The Attorney General is required to refer calls received on the whistleblower hotline to the appropriate government authority for review and possible investigation. During the initial review of such a call, the Attorney General or appropriate government agency shall hold in confidence information disclosed through the whistleblower hotline, including the identity of the caller disclosing the information and the employer identified by the caller. (Labor Code Section 1102.7.)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS: According to the author, “the greatest innovations happen when our brightest minds have the resources they need and the freedom to speak their minds. California’s leadership on AI is more critical than ever as the new federal Administration proceeds with shredding the guardrails meant to keep Americans safe from the known and foreseeable risks that advanced AI systems present.” The author contends that SB 53 will provide “a multipronged approach to encourage responsible AI innovation.” The bill will ensure that “employees of frontier model labs are not retaliated against for reporting their belief that a developer’s activities pose a critical risk . . . or has made false or misleading statements about its management of critical risk.” The bill would establish “a process to create a public cloud-computing cluster (CalCompute) that will conduct research into the safe and secure deployment of large-scale artificial intelligence (AI) models.” SB 53, the author concludes, will allow “California to continue to lead in this space and to demonstrate that safety does not stifle success.”
Promise and peril: the Little Hoover Commission report. In December, 2024, California’s Little Hoover Commission issued a report entitled, “Artificial Intelligence and California State Government.” The report began by noting that artificial intelligence (AI) is a “powerful force that has the potential to fundamentally transform the way we live . . . [but] in the absence of safeguards, AI can also expose us to risks of bias, misinformation, and significant economic dislocation.” The report added that the “rewards and risks are exacerbated” by new and developing forms of “generative AI,” which can imitate human thinking and produce novel content, including texts and images. The Little Hoover Commission set forth nine recommendations with the “unifying aim . . . to empower state workers, researchers, and others dedicated to serving the public good . . . to harness AI responsibly and effectively, driving innovation within state operations and beyond.” (Little Hoover Commission, Artificial Intelligence and California State Government, Report #284, December 9, 2024, p.3.)
The first of the Commission’s nine recommendations called upon California to “develop its own state-run, cloud-based AI computing center to provide affordable, scalable access to compute power, AI tools, and datasets for state agencies, academic institutions, non-profits, and startups.” The report stressed that cloud computing has become an essential resource because it provides the levels of “compute power” necessary to complete such “resource-intensive tasks like AI model training and analysis of large datasets.” Specifically, the Commission recommended that California should invest in and develop its own cloud-based computing platform, which would be owned and operated by the state. “Such an initiative,” the Commission maintained, “would not only meet the state’s own AI needs but also serve academic institutions, non-profits, and startups.” Indeed, it could even generate state revenue “by selling access to other governments and private entities.” (Id., pp. 11-14.) The goal of this state owned and operated system is to “democratize” access to AI resources and provide a space for public innovation and development that is independent of large technology companies. Corporate players like Meta and Google do extraordinary research and development, but their obligations may run more toward shareholders than the public interest. This is not to criticize these companies, for they are only doing what private companies must do. However, a state-owned and operated system could promote development more closely attuned to the public interest.
The first section of this bill does precisely what the Little Hoover Commission report called for: create a state-run, cloud-based computer cluster in order to advance the development and deployment of AI that is safe, ethical, equitable, and sustainable. It does this by establishing, within the Government Operations Agency (GOA), a fourteen-member consortium that will develop a framework for the creation of a “public cloud computing cluster” to be known as “CalCompute.” The bill requires the consortium to make “reasonable efforts” to ensure that CalCompute is established within the University of California to the extent possible. GOA must submit a report that contains that framework to the Legislature on or before January 1, 2027. At that point the consortium would dissolve and the Legislature, presumably, will make an appropriation to get CalCompute up and running within the University of California.
Whistleblower protection for disclosure of “critical risks.” The second section of this bill – and the one most relevant to this Committee – seeks to provide whistleblower protections to employees of AI developers. Existing law provides all employees and most contractors with whistleblower protection by prohibiting an employer from making or enforcing any rule, regulation, or policy that prevents an employee from disclosing information to any public body if the employee has a reasonable belief that the information discloses a violation of a law and regulation. Existing law also prohibits an employer, or an employer’s agent, from retaliating against any employee who exercises their right to disclose this information. While Labor Code Section 1102.5 protects private employees from retaliation, California’s Whistleblower Protection Act (Government Code 8547 et seq.) protects state employees. The law protecting state employees is somewhat broader than the law protecting private employees. While private employees are only protected if they disclose unlawful activity, state employees may additional disclose information that exposes waste or other government misconduct, whether technically unlawful or not.
According to the supporters of this bill, the existing whistleblower protection law for private employees is inadequate for employees of AI developers because it requires that the employer violates a law or regulation. However, as the author and supporters of this bill persuasively point out, AI development is so new and rapidly developing that an employer’s activity may not violate any existing law or regulation. Therefore, rather than relying on existing law, which requires a violation of law, this bill would prohibit an AI developer from making or enforcing any rule or contract provision that prohibits an employee from disclosing information about any developer activity that creates a “critical risk,” whether that critical risk constitutes a legal violation or not. It would also protect an employee who discloses false statements or representations about a known risk. Unfortunately, the bill in print defines “critical risk” in such a restrictive manner that, realistically, it may not protect any employee disclosures.
The problematic definition of “critical risk.” In order to show the overly restrictive nature of the bill’s definition, it may be useful to quote the definition directly and in its entirety:
(b) “Critical risk” means a foreseeable and material risk that a developer’s development, storage, or deployment of a foundation model will result in the death of, or serious injury to more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property, through any of the following: [Emphasis added.]
(1) The creation and release of a chemical, biological, radiological, or nuclear weapon.
(2) A cyberattack.
(3) A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
(4) A foundation model evading the control of its developer or user.
It is not entirely clear why the AI developer’s activity, in order to be the subject of a protected disclosure, must result in the death or serious injury of “more than 100” people, as opposed to 50 or even one. More problematic, not only must the development or deployment of the AI model result in the death or serious injury of more than 100 people to be the subject of a protected disclosure, those deaths or injuries must also occur “through” one of the four means listed as paragraphs (1) through (4), above. If the development or deployment resulted in 500 deaths, but did not fall in one those four categories, disclosing the activity would not be protected. On the other hand, if the result came about through one those categories (e.g. releasing a biological weapon), disclosure would not be protected if the activity could result less than 100 deaths. Moreover, it is not clear how an employee could possibly predict or estimate the number of deaths or serious injuries that could occur.
As a practical matter, it may be that any development or deployment of an AI model that caused more than one hundred deaths due to the “release of a chemical, biological, radiological or nuclear weapon,” or the loss of $1 billion due to a “cyberattack,” would necessarily entail the violation of some law, and would therefore be protected under the existing whistleblower statute. But if this so, it is not clear how this bill adds anything to existing law. The author’s office and sponsors have informed the Committee that their primary aim, at this point, is to protect disclosures of potentially catastrophic events. But given that there is no way to predict the number of deaths under these scenarios, it is unclear why the acts listed in paragraphs (1) through (4) must be coupled with the arbitrary requirement that the activity result in 100 or more deaths or serious injuries: either types of outcomes would seem to be sufficiently grave to be a “critical risk” all on their own.
Possible amendments the author may wish to consider in the next Committee, if the bill moves forward. If the intent of the bill is only to protect disclosures about activity that will result in more than 100 deaths or serious injuries – and only when those deaths or injuries come about “through” one of enumerated items – then the language of the bill can stand as it is.
However, if the intent is to protect disclosures about activities that threaten any human life, regardless of the form that the threat takes, then the author may wish to consider the following amendment to the definition of “critical risk:”
Possible Amendment #1: On page 5 starting on line 19 change the definition of “critical risk” as follows:
(b) “Critical risk” means a foreseeable and material risk that a developer’s development, storage, or deployment of a foundation model will result in the death of, or serious injury to, one or more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property. through any of the following:
(1) The creation and release of a chemical, biological, radiological, or nuclear weapon.
(2) A cyberattack.
(3) A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
(4) A foundation model evading the control of its developer or user.
Possible Amendment #2: If the intent of the bill is to protect disclosures relating to potentially catastrophic events suggested in the enumerated items – where the number of deaths is not predictable, or possibly even relevant – then the author may wish to consider the following amendment:
(b) “Critical risk” means a foreseeable and material risk that a developer’s development, storage, or deployment of a foundation model could result in any of the following: will result in the death of, or serious injury to more than 100 people or more than one billion dollars ($1,000,000,000) in damage to rights in money or property, through any of the following:
(1) The creation and release of a chemical, biological, radiological, or nuclear weapon.
(2) A cyberattack.
(3) A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
(4) A foundation model evading the control of its developer or user.
ARGUMENTS IN SUPPORT: The co-sponsors – Encode, Economic Security California Action, and Secure AI Project – along with a broad coalition supporting the development of safe and responsible AI, write in support:
Large AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The recent International Scientific Report on the Safety of Advanced AI, supported by over 30 countries including the United States, captures the consensus view that “...evidence of additional risks is gradually emerging. These include risks such as large-scale labor market impacts, AI-enabled hacking or biological attacks,” and other serious risks. Advanced AI is mostly unregulated, and these risks are currently being managed by companies themselves.
AI researchers and engineers will likely be the first to know if a company’s product poses a significant risk to public safety, but company contracts may prevent them from reporting concerns. In general, companies offer or require non-disclosure and non-disparagement agreements that are quite broad, often limiting nearly all disclosures that are not explicitly protected by law. These agreements in practice can prevent employees from sharing a good faith public safety concern with a government authority, which could chill employees’ disclosure rights when that is their best or only course of action.
By requiring companies to have an internal process where employees can report dangers, SB 53 would improve information flow and in turn enhance companies’ awareness of serious dangers.
SB 53 would also carefully expand existing whistleblower protections, which is essential for the AI industry. If internal reporting fails to adequately address an identified risk, employees need an option to turn to government authorities. In more established industries, safety hazards are likely to violate an existing law or regulation, and disclosure of those hazards would be protected under existing law. However, as a nearly unregulated industry, dangerous behavior at AI companies might not be illegal today and disclosure is not protected in the same manner. SB 53 addresses this imbalance by expanding essentially the same protections from retaliation already found for reporting violations of the law to material risks of serious catastrophic harm, as well as false statements about company safety practices. . .
In addition to whistleblower protections, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research and CalCompute would make those resources more accessible to California’s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting, in particular, smaller startups for a healthier innovation ecosystem. We support this groundbreaking effort, which would advance and democratize AI research in California.
REGISTERED SUPPORT / OPPOSITION:
Support
AI for Animals
AI Futures Project
AI Lab Watch
AI Policy Tracker
All Girls Allowed
Apart Research
Berkeley Existential Risk Initiative
Center for Ai and Digital Policy
Center for Ai Policy
Center for Human-Compatible AI
Children's Advocacy Institute, University of San Diego School of Law
Depict.AI
District Council of Iron Workers of the State of California and Vicinity
Earnings Stream LLC
Economic Security California Action
Elicit
Encode
Eon Systems
Indivisible California State Strong
Little Hoover Commission
Momentum
Nonlinear
Oakland Privacy
Redwood Research
Safe AI Future
Secure AI Future
Secure AI Project
Tech Oversight California
The Midas Project
The Signals Network
Trevi Digital Assets Fund
University of California
Support if amended (to prior version)
Chamber of Progress
Opposition
None on file
Analysis Prepared by: Tom Clark / JUD. / (916) 319-2334
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 53 |
THIRD READING
Bill No: SB 53
Author: Wiener (D)
Amended: 5/23/25
Vote: 21
SENATE GOVERNMENTAL ORG. COMMITTEE: 13-0, 3/25/25
AYES: Padilla, Valladares, Archuleta, Ashby, Blakespear, Cervantes, Hurtado, Jones, Ochoa Bogh, Richardson, Rubio, Wahab, Weber Pierson
NO VOTE RECORDED: Dahle, Smallwood-Cuevas
SENATE JUDICIARY COMMITTEE: 13-0, 4/8/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SUBJECT: CalCompute: foundation models: whistleblowers
SOURCE: Economic Security California Action
Encode
Secure AI Project
DIGEST:
This bill (1) establishes a consortium within the Government Operations Agency (GovOps) and tasks the consortium with developing a framework for the creation of a public cloud computing cluster that benefits the public, to be known as “CalCompute,” as specified. This bill (2) establishes whistleblower protections for employees working with specified artificial intelligence (AI) models when reporting on critical risks, as specified.
ANALYSIS:
Existing law:
- Establishes the California Department of Technology, within GovOps, and requires the department to conduct a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency, as specified.
- Prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill:
- Establishes a consortium, within GovOps, and requires the consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of AI that is safe, ethical, equitable, and sustainable, as specified.
- Requires the consortium to make reasonable efforts to ensure that CalCompute is established within the University of California (UC) to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following: a fully owned and hosted cloud platform; necessary human expertise to operate and maintain the platform; necessary human expertise to support, train, and facilitate the use of CalCompute, as specified.
- Requires GovOps, on or before January 1, 2027, to submit a report from the consortium to the Legislature with the framework developed pursuant to the above provisions for the creation and operation of CalCompute, as specified.
- Requires the consortium to, consistent with state constitutional law, consist of 14 members, as specified; and, provides that if CalCompute is established within the UC, the UC may receive private donations for the purpose of implementing CalCompute.
- Prohibits a developer, as defined, from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or from retaliating against an employee for disclosing, information to the Attorney General (AG), a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, as specified.
- Prohibits a developer from entering into a contract that prevents an employee from making a disclosure, as specified.
- Prohibits a developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the developer related to the assessment, management, or addressing of critical risk, or an employee of that organization or entity, from disclosing information to the AG, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloser either of the following: the developer’s activities pose a critical risk; the developer has made a false or misleading statement about its management of critical risk.
- Authorizes an employee, as defined, to use the whistleblower hotline maintained by the AG to make reports described above.
- Requires a developer to provide a clear notice to all employees of their rights and responsibilities under this bill, as specified.
- Requires a developer to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk, or that the developer has made a false or misleading statement about its management of critical risk, as specified.
- Requires the disclosures and responses of the process required by this bill to be shared with officers and directors of the developer at least once each quarter, as specified.
- Authorizes the court to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
- Provides that in a civil action brought pursuant to this bill, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this bill, as specified.
- Provides that in a civil action or administrative proceeding brought pursuant to this bill, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
- Provides, that upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
- Provides that in addition to any harm resulting directly from a violation of this bill, the court shall consider the chilling effect on other employees asserting their rights under this bill in determining whether temporary injunctive relief is just and proper.
- Requires appropriate injunctive relief be issued on a showing that reasonable cause exists to believe a violation has occurred.
- Requires an order authorizing temporary injunctive relief to remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review, as specified, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
- Provides that injunctive relief granted pursuant to this bill shall not be stayed pending appeal, as specified.
- Specifies that the remedies provided by this bill are cumulative to each other and the remedies or penalties available under all other laws of this state.
Background
Author Statement. According to the author’s office, “the greatest innovations happen when our brightest minds have the resources they need and the freedom to speak their minds. California’s leadership on AI is more critical than ever as the new federal Administration proceeds with shredding the guardrails meant to keep Americans safe from the known and foreseeable risks that advanced AI systems present.”
Further, “Senate Bill 53 recognizes that we need a multipronged approach to encourage responsible AI innovation. It does so by: 1. Ensuring employees of frontier model labs are not retaliated against for reporting their belief that a developer’s activities pose a critical risk, as defined, or has made false or misleading statements about its management of critical risk; and, 2. Establishing a process to create a public cloud-computing cluster (CalCompute) that will conduct research into the safe and secure deployment of large-scale AI models.”
Cluster Computing. Cluster computing is a type of computing where multiple computers are connected to work together as a single system. The term “cluster” refers to the network of linked computer systems programmed performing the same task. Computing clusters typically consist of servers, workstations, and personal computers that communicate over a local area network or a wide area network. Cluster computing is a kind of distributed computing, a type of computing that links computers together on a network to perform a computational task, increase computational power and function as a single computer. Each computer, or “node,” in a cluster network has an operating system and a central processing unit core that handles the tasks required for the software to run properly.
CalCompute. This bill requires GovOps to establish a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute.” By January 1, 2027, GovOps must submit to the Legislature the consortium’s report with the framework. This bill requires the framework for CalCompute to advance the development and deployment of AI that is safe, ethical, equitable, and sustainable by fostering research and innovation that benefits the public and enabling equitable innovation by expanding access to computational resources.
The consortium will be consist of eight members selected by the Secretary of GovOps, and three each by the President pro Tempore of the Senate and the Speaker of the Assembly. Specifically, this bill requires the 14 members be selected among representatives of the UC and other public and private academic research institutions and national laboratories; representatives of impacted workforce labor organizations; representatives of stakeholder groups with relevant expertise and experience; experts in technology and AI to provide technical assistance; and personnel from other relevant departments and agencies, as necessary. This bill does not establish CalCompute, however, it does specify that if CalCompute is eventually established within the UC, the UC may receive private donations for the purposes of implementing CalCompute.
Frontier Models and Safety. Frontier AI models are the cutting-edge class of “foundation” or general-purpose AI systems. These models have the ability to perform a wide variety of tasks – from natural language understanding to code generation and image creation – and are trained with incredible amounts of compute, data, energy, and water. These models are distinguished by their breadth of functionality and the scale of resources required to develop them (often costing hundreds of millions or billions of dollars). Their ability to produce powerful new capabilities potentially pose severe risks to safety or security if misused. Model derivatives are any downstream version of a frontier model created by modifying or adapting the original model’s weights, architecture, or outputs for specialized purposes and applications.
Critical Risk and AI Whistleblowers. This bill strengthens whistleblower protections for employees of frontier AI laboratory companies whose activities developing frontier AI pose a critical risk, as defined. These protections apply to companies that have trained at least one foundation model with a quantity of computational power that costs at least $100 million. This bill prevents developers of these frontier models from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the AG, a federal authority, a person with authority over the employee, or another employee, as specified, if the employee has reasonable cause to believe that the information discloses that the developer’s activities pose a critical risk or the developer has made false or misleading statements about its management of critical risk.
A developer must also establish a reasonable internal process for employees to anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk. The process must include a monthly update to the person who made the disclosure regarding the status of the developer’s investigation of the disclosure and the actions taken by the developer in response to the disclosure.
Violations may be remedied through civil or administrative actions and courts are authorized to award a prevailing plaintiff injunctive relief and reasonable attorney’s fees. In a civil action, this bill alters the relevant burdens. Once it has been demonstrated by a preponderance of the evidence that a relevant protected activity was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in the protected activities. Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
Related/Prior Legislation
SB 1047 (Wiener, 2024) would have required developers of powerful AI models, as defined, and those providing the computer power to train such models to put appropriate safeguards and policies in place to prevent critical harms, as defined. The bill ould have established a state entity to oversee the development of these models and called for the creation of a consortium to develop a framework for a public cloud computing cluster. SB 1047 was vetoed by Governor Newsom.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee, GovOps estimates costs ranging from $2.5 million to $7.5 million, depending on the scale of the CalCompute system framework the consortium is tasked with developing (General Fund). Costs include staffing a team to develop and manage the project, which would include contracting with experts and other staff time for research, strategic planning, and development.
Unknown, potentially significant workload cost pressures to the courts to adjudicate alleged violations of this measure (Trial Court Trust Fund, General Fund).
SUPPORT: (Verified 5/23/2025)
Economic Security California Action (Co-source)
Encode (Co-source)
Secure AI Project (Co-source)
Ai for Animals
AI Futures Project
AI Lab Watch
AI Policy Tracker
Apart Research
BERI
Center for AI and Digital Policy
Center for Human-Compatible Artificial Intelligence, UC Berkeley
Children’s Advocacy Institute, University of San Diego School of Law
Depict.ai
EarningsStream LLC
Elicit
Eon Systems
Indivisible CA: StateStrong
Little Hoover Commission
Momentum
Nonlinear
Oakland Privacy
Redwood Research
Safe AI Future
The Midas Project
Trevi Digital Assets Fund
University of California
Youth Leadership Institute
OPPOSITION: (Verified 5/23/2025)
None received
ARGUMENTS IN SUPPORT: The co-sources of the bill and a coalition of supporters jointly write in support that, “[l]arge AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The recent International Scientific Report on the Safety of Advanced AI, supported by over 30 countries including the United States, captures the consensus view that ‘…evidence of additional risks is gradually emerging. These include risks such as large-scale labor market impacts, AI-enabled hacking or biological attacks,’ and other serious risks. Advanced AI is mostly unregulated, and these risks are currently being managed by companies themselves.”
Prepared by: Brian Duke / G.O. / (916) 651-1530
5/27/25 11:29:53
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 53 |
THIRD READING
Bill No: SB 53
Author: Wiener (D)
Amended: 5/23/25
Vote: 21
SENATE GOVERNMENTAL ORG. COMMITTEE: 13-0, 3/25/25
AYES: Padilla, Valladares, Archuleta, Ashby, Blakespear, Cervantes, Hurtado, Jones, Ochoa Bogh, Richardson, Rubio, Wahab, Weber Pierson
NO VOTE RECORDED: Dahle, Smallwood-Cuevas
SENATE JUDICIARY COMMITTEE: 13-0, 4/8/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SUBJECT: CalCompute: foundation models: whistleblowers
SOURCE: Economic Security California Action
Encode
Secure AI Project
DIGEST:
This bill establishes a consortium within the Government Operations Agency (GovOps) and tasks the consortium with developing a framework for the creation of a public cloud computing cluster that benefits the public, to be known as “CalCompute,” as specified. Additionally, this bill establishes whistleblower protections for employees working with specified artificial intelligence (AI) models when reporting on critical risks, as specified.
ANALYSIS:
Existing law:
- Establishes the California Department of Technology (CDT), within GovOps, and requires the department to conduct a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency, as specified.
- Prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
This bill:
- Establishes a consortium, within GovOps, and requires the consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of AI that is safe, ethical, equitable, and sustainable, as specified.
- Requires the consortium to make reasonable efforts to ensure that CalCompute is established within the University of California (UC) to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following: a fully owned and hosted cloud platform; necessary human expertise to operate and maintain the platform; necessary human expertise to support, train, and facilitate the use of CalCompute, as specified.
- Requires GovOps, on or before January 1, 2027, to submit a report from the consortium to the Legislature with the framework developed pursuant to the above provisions for the creation and operation of CalCompute, as specified.
- Requires the consortium to, consistent with state constitutional law, consist of 14 members, as specified; and, provides that if CalCompute is established within the UC, the UC may receive private donations for the purpose of implementing CalCompute.
- Prohibits a developer, as defined, from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that prevents an employee from disclosing, or from retaliating against an employee for disclosing, information to the Attorney General (AG), a federal authority, a person with authority over the employee, or another employee who has authority to investigate, discover, or correct the reported issue, as specified.
- Prohibits a developer from entering into a contract that prevents an employee from making a disclosure, as specified.
- Prohibits a developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the developer related to the assessment, management, or addressing of critical risk, or an employee of that organization or entity, from disclosing information to the AG, a federal authority, or the developer if the organization, entity, or individual has reasonable cause to believe that the information discloser either of the following: the developer’s activities pose a critical risk; the developer has made a false or misleading statement about its management of critical risk.
- Authorizes an employee, as defined, to use the whistleblower hotline maintained by the AG to make reports described above.
- Requires a developer to provide a clear notice to all employees of their rights and responsibilities under this bill, as specified.
- Requires a developer to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk, or that the developer has made a false or misleading statement about its management of critical risk, as specified.
- Requires the disclosures and responses of the process required by this bill to be shared with officers and directors of the developer at least once each quarter, as specified.
- Authorizes the court to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
- Provides that in a civil action brought pursuant to this bill, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this bill, as specified.
- Provides that in a civil action or administrative proceeding brought pursuant to this bill, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
- Provides, that upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
- Provides that in addition to any harm resulting directly from a violation of this bill, the court shall consider the chilling effect on other employees asserting their rights under this bill in determining whether temporary injunctive relief is just and proper.
- Requires appropriate injunctive relief be issued on a showing that reasonable cause exists to believe a violation has occurred.
- Requires an order authorizing temporary injunctive relief to remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review, as specified, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
- Provides that injunctive relief granted pursuant to this bill shall not be stayed pending appeal, as specified.
- Specifies that the remedies provided by this bill are cumulative to each other and the remedies or penalties available under all other laws of this state.
Background
Author Statement. According to the author’s office, “the greatest innovations happen when our brightest minds have the resources they need and the freedom to speak their minds. California’s leadership on AI is more critical than ever as the new federal Administration proceeds with shredding the guardrails meant to keep Americans safe from the known and foreseeable risks that advanced AI systems present.” Further, “Senate Bill 53 recognizes that we need a multipronged approach to encourage responsible AI innovation. It does so by: 1. Ensuring employees of frontier model labs are not retaliated against for reporting their belief that a developer’s activities pose a critical risk, as defined, or has made false or misleading statements about its management of critical risk; and, 2. Establishing a process to create a public cloud-computing cluster (CalCompute) that will conduct research into the safe and secure deployment of large-scale AI models.”
Cluster Computing. Cluster computing is a type of computing where multiple computers are connected to work together as a single system. The term “cluster” refers to the network of linked computer systems programmed performing the same task. Computing clusters typically consist of servers, workstations, and personal computers that communicate over a local area network or a wide area network. Cluster computing is a kind of distributed computing, a type of computing that links computers together on a network to perform a computational task, increase computational power and function as a single computer. Each computer, or “node,” in a cluster network has an operating system and a central processing unit core that handles the tasks required for the software to run properly.
CalCompute. This bill requires GovOps to establish a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute.” By January 1, 2027, GovOps must submit to the Legislature the consortium’s report with the framework. This bill requires the framework for CalCompute to advance the development and deployment of AI that is safe, ethical, equitable, and sustainable by fostering research and innovation that benefits the public and enabling equitable innovation by expanding access to computational resources.
The consortium will be consist of eight members selected by the Secretary of GovOps, and three each by the President pro Tempore of the Senate and the Speaker of the Assembly. Specifically, this bill requires the 14 members be selected among representatives of the UC and other public and private academic research institutions and national laboratories; representatives of impacted workforce labor organizations; representatives of stakeholder groups with relevant expertise and experience; experts in technology and AI to provide technical assistance; and personnel from other relevant departments and agencies, as necessary. This bill does not establish CalCompute, however, it does specify that if CalCompute is eventually established within the UC, the UC may receive private donations for the purposes of implementing CalCompute.
Frontier Models and Safety. Frontier AI models are the cutting-edge class of “foundation” or general-purpose AI systems. These models have the ability to perform a wide variety of tasks – from natural language understanding to code generation and image creation – and are trained with incredible amounts of compute, data, energy, and water. These models are distinguished by their breadth of functionality and the scale of resources required to develop them (often costing hundreds of millions or billions of dollars). Their ability to produce powerful new capabilities potentially pose severe risks to safety or security if misused. Model derivatives are any downstream version of a frontier model created by modifying or adapting the original model’s weights, architecture, or outputs for specialized purposes and applications.
Critical Risk and AI Whistleblowers. This bill strengthens whistleblower protections for employees of frontier AI laboratory companies whose activities developing frontier AI pose a critical risk, as defined. These protections apply to companies that have trained at least one foundation model with a quantity of computational power that costs at least $100 million. Specifically, this bill prevents developers of these frontier models from making, adopting, enforcing, or entering into any rule, regulation, policy, or contract that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the AG, a federal authority, a person with authority over the employee, or another employee, as specified, if the employee has reasonable cause to believe that the information discloses that the developer’s activities pose a critical risk or the developer has made false or misleading statements about its management of critical risk.
A developer must also establish a reasonable internal process for employees to anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk. The process must include a monthly update to the person who made the disclosure regarding the status of the developer’s investigation of the disclosure and the actions taken by the developer in response to the disclosure.
Violations may be remedied through civil or administrative actions and courts are authorized to award a prevailing plaintiff injunctive relief and reasonable attorney’s fees. In a civil action, this bill alters the relevant burdens. Once it has been demonstrated by a preponderance of the evidence that a relevant protected activity was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in the protected activities. Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
Related/Prior Legislation
SB 1047 (Wiener, 2024) would have required developers of powerful AI models, as defined, and those providing the computer power to train such models to put appropriate safeguards and polici4es in place to prevent critical harms, as defined. Would have established a state entity to oversee the development of these models and called for the creation of a consortium to develop a framework for a public cloud computing cluster. (Vetoed by Governor Newsom)
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee, GovOps estimates costs ranging from $2.5 million to $7.5 million, depending on the scale of the CalCompute system framework the consortium is tasked with developing (General Fund). Costs include staffing a team to develop and manage the project, which would include contracting with experts and other staff time for research, strategic planning, and development.
Unknown, potentially significant workload cost pressures to the courts to adjudicate alleged violations of this measure (Trial Court Trust Fund, General Fund).
SUPPORT: (Verified 5/23/2025)
Economic Security California Action (Co-source)
Encode (Co-source)
Secure AI Project (Co-source)
AI for Animals
AI Futures Project
AI Lab Watch
AI Policy Tracker
Apart Research
BERI
Center for AI and Digital Policy
Center for Human-Compatible Artificial Intelligence, UC Berkeley
Children’s Advocacy Institute, University of San Diego School of Law
Depict.ai
EarningsStream LLC
Elicit
Eon Systems
Indivisible CA: StateStrong
Little Hoover Commission
Momentum
Nonlinear
Oakland Privacy
Redwood Research
Safe AI Future
The Midas Project
Trevi Digital Assets Fund
University of California
Youth Leadership Institute
OPPOSITION: (Verified 5/23/2025)
None received
ARGUMENTS IN SUPPORT: The co-sources of this bill and a coalition of supporters jointly write in support that, “[l]arge AI developers are developing increasingly advanced AI systems. We are excited about the potential for these systems to drive improvements in education, science, provisioning of public services, and more. At the same time, large AI developers themselves warn that their AI systems could pose serious risks, which they have voluntarily committed to addressing. The recent International Scientific Report on the Safety of Advanced AI, supported by over 30 countries including the United States, captures the consensus view that ‘…evidence of additional risks is gradually emerging. These include risks such as large-scale labor market impacts, AI-enabled hacking or biological attacks,’ and other serious risks. Advanced AI is mostly unregulated, and these risks are currently being managed by companies themselves.”
Prepared by: Brian Duke / G.O. / (916) 651-1530
5/27/25 12:28:41
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2025 - 2026 Regular Session
SB 53 (Wiener) - CalCompute: foundation models: whistleblowers
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
SB 53 establishes a consortium within the Government Operations Agency (GovOps) and tasks the consortium with developing a framework for the creation of a public cloud computing cluster that benefits the public, to be known as “CalCompute,” as specified. SB 53 also establishes whistleblower protections for employees working with specified artificial intelligence (AI) models when reporting on critical risks, as specified.
*********** ANALYSIS ADDENDUM – SUSPENSE FILE ***********
The following information is revised to reflect amendments
adopted by the committee on May 23, 2025
Fiscal Impact:
- GovOps estimates costs ranging from $2.5 million to $7.5 million, depending on the scale of the CalCompute system framework the consortium is tasked with developing (General Fund). Costs include staffing a team to develop and manage the project, which would include contracting with experts and other staff time for research, strategic planning, and development.
- Unknown, potentially significant workload cost pressures to the courts to adjudicate alleged violations of this measure (Trial Court Trust Fund, General Fund).
Author Amendments:
- Modify the definitions of “employee” and “foundation model.”
- Define “labor contractor” as an individual or entity that supplies, either with or without a contract, a client employer with workers to perform labor within the client employer’s usual course of business.
- Expand whistleblower protections for an employee disclosing information to a person with authority over the employee.
- Additionally prohibit a developer from entering into a contract that prevents an employee from making a disclosure protected under Labor Code (LC) § 1102.5.
- Additionally prohibit a developer from making, adopting, enforcing, or entering into a rule, regulation, policy, or contract that would prevent an organization or entity that provides goods or services to the developer related to the assessment, management, or addressing of critical risk, or an employee of that organization or entity, from disclosing information to the Attorney General, a federal authority, or the developer if there is reasonable cause to believe that the information discloses either of the following:
- The developer’s activities pose a critical risk.
- The developer has made a false or misleading statement about its management of critical risk.
- Provide that the bill does not impair or limit the applicability of LC § 1102.5.
- Provide the remedies provided by the bill are cumulative to each other and the remedies or penalties available under all other laws of this state.
- Make other technical, clarifying, and conforming changes.
-- END --
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2025 - 2026 Regular Session
SB 53 (Wiener) - CalCompute: foundation models: whistleblowers
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
SB 53 establishes a consortium within the Government Operations Agency (GovOps) and tasks the consortium with developing a framework for the creation of a public cloud computing cluster that benefits the public, to be known as “CalCompute,” as specified. SB 53 also establishes whistleblower protections for employees working with specified artificial intelligence (AI) models when reporting on critical risks, as specified.
Fiscal Impact:
GovOps estimates costs ranging from $2.5 million to $7.5 million, depending on the scale of the CalCompute system framework the consortium is tasked with developing (General Fund). Costs include staffing a team to develop and manage the project, which would include contracting with experts and other staff time for research, strategic planning, and development.
Background:
Cluster computing is a type of computing where multiple computers are connected to work together as a single system. The term “cluster” refers to the network of linked computer systems performing the same task. Computing clusters typically consist of servers, workstations, and personal computers that communicate over a local area network or a wide area network. Cluster computing is a kind of distributed computing, a type of computing that links computers together on a network to perform a computational task, increase computational power and function as a single computer. Each computer, or “node,” in a cluster network has an operating system and a central processing unit core that handles the tasks required for the software to run properly. Because of its high performance and high availability, cluster computing has many applications, including cloud computing, AI, scientific research, and big data analytics.
Proposed Law:
- Establishes a consortium within GovOps. Tasks the consortium with developing a framework for the creation of a public cloud computing cluster, to be known as “CalCompute,” that advances the development and deployment of AI that is safe, ethical, equitable, and sustainable by fostering research and innovation that benefits the public and enabling equitable innovation by expanding access to computational resources.
- Provides the consortium consist of 14 members selected from: representatives of the University of California (UC) and other public and private academic research institutions; representatives of impacted workforce labor organizations; representatives of stakeholder groups; experts in technology and AI; and personnel from other relevant departments and agencies. Specifies eight consortium members be selected by GovOps. Specifies the President pro Tempore of the Senate and the Speaker of the Assembly each select three members.
- Requires the consortium to make reasonable efforts to ensure that CalCompute is established within the UC to the extent possible.
- Requires CalCompute to include, but not be limited to, a fully owned and hosted cloud platform and necessary human expertise to operate and maintain the platform through training and other support.
- Requires the consortium to operate in accordance with all relevant labor and workforce laws and standards.
- By January 1, 2027, requires GovOps to submit a report from the consortium to the Legislature with the framework developed for the creation and operation of CalCompute. Requires the report to include a landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure; an analysis of the cost to the state to build and main CalCompute; recommendations for CalCompute governance and structure; recommendations for CalCompute parameters; an analysis of the state’s technology workforce; a description of proposed partnerships for CalCompute; and recommendations regarding prioritizing the use of the current public sector workforce to create and manage CalCompute.
- Requires the consortium to be dissolved upon submission of the report.
- Authorizes the UC to receive private donations to implement CalCompute if CalCompute is established within the UC.
- Provides that establishment of the consortium become operative only upon an appropriation in a budget act or other measure.
- Establishes whistleblower protections for employees working with specified AI models when reporting critical risks, as specified.
- Establishes definitions.
Related Legislation:
SB 579 (Padilla, 2025) requires the Secretary of GovOps to appoint a mental health and AI working group to evaluate the role and risks of AI in mental health settings and issue a report to the Legislature on its findings and recommendations, as specified. SB 579 is pending in this committee.
-- END --
Senator Thomas Umberg, Chair
2025-2026 Regular Session
Bill No: SB 53 (Author:Wiener)
Version: March 27, 2025
Hearing Date: April 8, 2025
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
CalCompute: foundation models: whistleblowers
DIGEST
This bill establishes a consortium tasked with developing a framework for a public cloud computing cluster that advances the ethical development and deployment of AI for the public good. This bill creates protections for whistleblowers working with specified AI models when reporting on “critical risks” and requires developers to provide processes for anonymous reporting of activities posing such risks.
EXECUTIVE SUMMARY
Large AI models have opened beneficial possibilities and breakthroughs in a variety of sectors, including healthcare, creative industries, education, business operations, and customer service. However, these models also pose significant risks to society, whether it is creation of chemical, biological, radiological, or nuclear (CBRN) weapons, utilization to carry out cyberattacks at scale, or evasion of the oversight and control of their developers or deployers. Given how complex and opaque these models can be and the lack of comprehensive regulatory oversight of their development, the need for stronger protections for experts working on these models who come forward to warn of such risks is clear.
This bill establishes baseline whistleblower protections for employees that disclose information to proper authorities or management where there is reasonable cause to believe that the information discloses activities on the part of a developer that poses a “critical risk” or that discloses misstatements about the management of critical risks. Developers are required to notify employees of these protections and to create internal reporting systems for anonymous disclosures of this sort. The bill also establishes a consortium tasked with developing a framework for a public cloud computing cluster, to be called “CalCompute,” that will advance ethical AI development for public benefit to counter the imbalance in access to AI infrastructure.
The bill is sponsored by Encode, Secure AI Project, and Economic Security California Action. It is supported by a number of advocacy organizations, including Oakland Privacy. No timely opposition was received by the Committee. The bill passed out of the Senate Governmental Organization Committee on a vote of 13 to 0.
PROPOSED CHANGES TO THE LAW
Existing law:
- Prohibits employers and any person acting on behalf of the employer from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of a law, as specified. Employers and their agents are also prohibited from retaliating against an employee for such conduct. (Lab. Code § 1102.5.)
- Requires the office of the Attorney General to maintain a whistleblower hotline to receive calls from persons who have information regarding possible violations of state or federal statutes, rules, or regulations, or violations of fiduciary responsibility by a corporation or limited liability company to its shareholders, investors, or employees. The Attorney General is required to refer calls received on the whistleblower hotline to the appropriate government authority for review and possible investigation. During the initial review of such a call, the Attorney General or appropriate government agency shall hold in confidence information disclosed through the whistleblower hotline, including the identity of the caller disclosing the information and the employer identified by the caller. (Lab. Code § 1102.7.)
- Establishes the California Department of Technology (CDT) within the Government Operations Agency (GovOps), which is tasked with conducting, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. (Gov. Code §§ 11545(a), 11546.45.5.)
- Provides that the perfecting of an appeal stays proceedings in the trial court upon the judgment or order appealed from or upon the matters embraced therein or affected thereby, including enforcement of the judgment or order, but the trial court may proceed upon any other matter embraced in the action and not affected by the judgment or order. When there is a stay of proceedings other than the enforcement of the judgment, the trial court shall have jurisdiction of proceedings related to the enforcement of the judgment as well as any other matter embraced in the action and not affected by the judgment or order appealed from. (Code Civ. Proc. § 916.)
This bill:
- Establishes within GovOps a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of AI that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
- Fostering research and innovation that benefits the public.
- Enabling equitable innovation by expanding access to computational resources.
- Provides that the consortium shall operate in accordance with all relevant labor and workforce laws and standards and make reasonable efforts to ensure that CalCompute is established within the University of California (UC) to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate the use of CalCompute.
- Requires GovOps, on or before January 1, 2027, to submit a report from the consortium to the Legislature with the framework for CalCompute. The report shall include all of the following elements:
- A landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure.
- An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
- Recommendations for the governance structure and ongoing operation of CalCompute.
- Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
- An analysis of the state’s technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
- A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
- Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
- Provides that the consortium shall, consistent with state constitutional law, consist of 14 members selected from specified fields and industries. Eight members are to be selected by GovOps, and the President pro Tempore of the Senate and the Speaker of the Assembly shall each select three members. The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties. The consortium shall be dissolved upon submission of the report.
- Provides that if CalCompute is established within the UC, the UC may receive private donations for the purposes of implementing CalCompute.
- Provides that the above provisions shall become operative only upon an appropriation in a budget act, or other measure, for the purposes of this section.
- Prohibits a developer from making, adopting, or enforcing a rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses either of the following:
- The developer’s activities pose a critical risk.
- The developer has made false or misleading statements about its management of critical risk.
- Defines “critical risk” as a foreseeable and material risk that a developer’s development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than $1 billion in damage to rights in money or property, through any of the following:
- The creation and release of a CBRN weapon.
- A cyberattack.
- A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
- A foundation model evading the control of its developer or user.
- Authorizes an employee to use the hotline described in Section 1102.7 of the Labor Code to make the above reports.
- Requires a developer to provide a clear notice to all employees of their rights and responsibilities hereunder, as specified.
- Requires a developer to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk, including a monthly update to the person who made the disclosure regarding the status of the developer’s investigation of the disclosure and the actions taken by the developer in response to the disclosure.
- Requires the disclosures and responses associated with the above process to be shared with officers and directors of the developer at least once each quarter, except as to any officer or director accused of wrongdoing.
- Authorizes civil actions to be brought for violations. The court is authorized to award reasonable attorney’s fees and injunctive relief to a plaintiff who brings a successful action for a violation of this section. In a civil action brought pursuant hereto, once it has been demonstrated by a preponderance of the evidence that an activity proscribed was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this section.
- Provides that, in a civil action or administrative proceeding brought pursuant hereto, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief. Upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
- Provides that, in addition to any harm resulting directly from a violation of this section, the court shall consider the chilling effect on other employees asserting their rights under this section in determining whether temporary injunctive relief is just and proper. Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
- Requires an order authorizing temporary injunctive relief to remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review, as provided, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation. Notwithstanding Section 916 of the Code of Civil Procedure, injunctive relief granted pursuant hereto shall not be stayed pending appeal.
COMMENTS
- Whistleblower protections
The powerful benefits and existential risks of AI technology, and particularly large generative AI models, is well documented. Despite a flurry of introduced legislation across the country, there is still not a comprehensive regulatory framework to ensure the safe and equitable development of this technology. This bill seeks to implement baseline protections for potential whistleblowers to ensure those closest to the development of this technology feel comfortable bringing concerns to the fore.
Whistleblower protections within companies developing AI foundation models are crucial given the extraordinary scale and impact of the technology. These protections serve as an essential safety valve in an industry where the potential consequences of unethical or dangerous development practices can affect millions, if not billions, of people.
These models increasingly power critical infrastructure across healthcare, finance, employment, education, and government services. When employees with direct knowledge of risks, harms, or unethical practices cannot safely speak up, dangerous systems may be deployed without proper safeguards or public awareness. Whistleblowers often represent the last line of defense when corporate incentives prioritize growth, profit, or competitive advantage over public welfare.
The technical complexity of foundation models creates significant information asymmetry between companies and outside regulators or the public. Most governmental bodies lack the specialized expertise to effectively evaluate these systems without insider information. This knowledge gap makes whistleblowers uniquely positioned to identify issues that might otherwise remain hidden from external reviewers. Their technical understanding allows them to recognize significant risks that non-specialists or outsiders might miss.
Given the regulatory vacuum in this industry, internal voices raising concerns about potential harms serve a crucial function in preventing the deployment of unsafe systems. Without protected channels for these voices, companies may face fewer checks on irresponsible development practices until after harm has occurred.
This bill prohibits a “developer” from making, adopting, or enforcing a rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the Attorney General, federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, if the employee has reasonable cause to believe that the information discloses that the developer’s activities pose a critical risk or that the developer has made false or misleading statements about its management of critical risk.
“Developer” is narrowly defined to include only those persons who have trained a foundation model with a quantity of computational power that costs at least $100 million. “Critical risk” is defined to mean a foreseeable and material risk that a developer’s development, storage, or deployment of a foundation model will result in the death of, or serious injury to, more than 100 people or more than $1 billion in damage to rights in money or property, through any of the following:
- The creation and release of a CBRN weapon.
- A cyberattack.
- A foundation model engaging in conduct, with limited human intervention, that would, if committed by a human, constitute a violation of the Penal Code that requires intent, recklessness, or gross negligence or the solicitation or aiding and abetting of that violation.
- A foundation model evading the control of its developer or user.
Developers are required to provide notice to all employees of their rights and responsibilities hereunder, including posting and displaying a notice to all employees of their rights, ensuring that any new employee receives equivalent notice, and ensuring that any employee who works remotely periodically receives an equivalent notice; or annually providing written notice to each employee of the employee’s rights hereunder and ensuring that the notice is received and acknowledged by all of those employees.
A developer must also establish a reasonable internal process for employees to anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk. The process must include a monthly update to the person who made the disclosure regarding the status of the developer’s investigation of the disclosure and the actions taken by the developer in response to the disclosure.
Violations may be remedied through civil or administrative actions and courts are authorized to award a prevailing plaintiff injunctive relief and reasonable attorney’s fees. In a civil action, the bill alters the relevant burdens. Once it has been demonstrated by a preponderance of the evidence that a relevant protected activity was a contributing factor in the alleged prohibited action against the employee, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in the protected activities. Appropriate injunctive relief shall be issued on a showing that reasonable cause exists to believe a violation has occurred.
- CalCompute
As industry races toward developing larger, more powerful AI models and seeks to commodify the seemingly infinite applications of AI, concerns are growing about the diminishing role that researchers, academic institutions, and more public-focused entities are playing in the development of AI. As reported by the Washington Post:
As such tech behemoths as Meta, Google and Microsoft funnel billions of dollars into AI, a massive resources gap is building with even the country’s richest universities. Meta aims to procure 350,000 of the specialized computer chips — called GPUs — that are essential to run the gargantuan calculations needed for AI models. In contrast, Stanford’s Natural Language Processing Group has 68 GPUs for all of its work.
To obtain the expensive computing power and data required to research AI systems, scholars frequently partner with tech employees. Meanwhile, tech firms’ eye-popping salaries are draining academia of star talent.
Big tech companies now dominate breakthroughs in the field. In 2022, the tech industry created 32 significant machine learning models, while academics produced three, a significant reversal from 2014, when the majority of AI breakthroughs originated in universities, according to a Stanford report.
Researchers say this lopsided power dynamic is shaping the field in subtle ways, pushing AI scholars to tailor their research for commercial use. Last month, Meta CEO Mark Zuckerberg announced that the company’s independent AI research lab would move closer to its product team, ensuring “some level of alignment” between the groups, he said.
“The public sector is now significantly lagging in resources and talent compared to that of industry,” said [Fei-Fei] Li, a former Google employee and the co-director of the Stanford Institute for Human-Centered AI. “This will have profound consequences because industry is focused on developing technology that is profit-driven, whereas public-sector AI goals are focused on creating public goods.”
…
As Silicon Valley races to build chatbots and image generators, it is drawing would-be computer science professors with high salaries and the chance to work on interesting AI problems. Nearly 70 percent of people with PhDs in AI end up in private industry compared with 21 percent of graduates two decades ago, according to a 2023 report.[1]
The bill seeks to address this by establishing a consortium with GovOps tasked with developing a framework for the creation of a public cloud computing cluster to be known as “CalCompute.” The explicit direction is for CalCompute to advance the development and deployment of AI that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public and enabling equitable innovation by expanding access to computational resources.
The bill requires that CalCompute include a fully owned and hosted cloud platform and the necessary human expertise to operate and maintain the platform and to support, train, and facilitate the use of it. A report shall be submitted to the Legislature by January 1, 2027 at which point the consortium will be dissolved.
- Stakeholder positions
According to the author:
The greatest innovations happen when our brightest minds have the resources they need and the freedom to speak their minds. California’s leadership on AI is more critical than ever as the new federal Administration proceeds with shredding the guardrails meant to keep Americans safe from the known and foreseeable risks that advanced AI systems present.
Senate Bill 53 recognizes that we need a multipronged approach to encourage responsible AI innovation. It does so by:
- Ensuring employees of frontier model labs are not retaliated against for reporting their belief that a developer’s activities pose a critical risk, as defined, or has made false or misleading statements about its management of critical risk; and
- Establishing a process to create a public cloud-computing cluster (CalCompute) that will conduct research into the safe and secure deployment of large-scale artificial intelligence (AI) models.
In doing this, SB 53 allows California to continue to lead in this space and to demonstrate that safety does not stifle success.
A coalition of organizations, including the three co-sponsors, Secure AI Project, Encode, and Economic Security California Action, write:
AI researchers and engineers will likely be the first to know if a company’s product poses a significant risk to public safety, but company contracts may prevent them from reporting concerns. In general, companies offer or require broad non-disclosure and non-disparagement agreements that are quite broad, often limiting nearly all disclosures that are not explicitly protected by law. These agreements in practice can prevent employees from sharing a good faith public safety concern with a government authority, which could chill employees’ disclosure rights when that is their best or only course of action.
An anonymous internal process would encourage whistleblowers to report safety issues when necessary and without the government needing to get involved. But employees at AI companies in today’s hypercompetitive markets may be concerned about reporting their concerns to their supervisors for fear of dismissal or retaliation. By requiring companies to have an internal process where employees can report dangers, SB 53 would improve information flow and in turn enhance companies’ awareness of serious dangers. Many companies already have internal processes, such as OpenAI, Anthropic, xAI, and Microsoft, so this requirement would codify what is an emerging industry norm.
SB 53 would also carefully expand existing whistleblower protections, which is essential for the AI industry. If internal reporting fails to adequately address an identified risk, employees need an option to turn to government authorities. In more established industries, safety hazards are likely to violate an existing law or regulation, and disclosure of those hazards would be protected under existing law. However, as a nearly unregulated industry, dangerous behavior at AI companies might not today be illegal and disclosure is not protected in the same manner. SB 53 addresses this imbalance by expanding essentially the same protections from retaliation already found for reporting violations of the law to material risks of serious catastrophic harm, as well as false statements about company safety practices.
SUPPORT
Economic Security California Action (sponsor)
Encode (sponsor)
Secure AI Project (sponsor)
AI Policy Tracker
Apart Research
Beri
Center for AI and Digital Policy
Center for AI Policy
Center for Human-Compatible AI
EarningsStream LLC
Indivisible CA: StateStrong
Nonlinear
Oakland Privacy
Porn Free Colorado
Redwood Research
Safe AI Future
OPPOSITION
None received
RELATED LEGISLATION
Pending Legislation: AB 1405 (Bauer-Kahan, 2025) establishes an enrollment process for AI auditors within GovOps. It creates a publicly accessible repository of AI auditors and requires that they adhere to minimum standards of transparency, confidentiality, and ethical conduct. It also provides for whistleblower protections in certain cases for employees employed by enrolled auditors. AB 1405 is currently in the Assembly Appropriations Committee.
Prior Legislation: SB 1047 (Wiener, 2024) would have required developers of powerful AI models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. It would have established a state entity to oversee the development of these models and called for the creation of a consortium to develop a framework for a public cloud computing cluster, as provided for by this bill.
Governor Newsom vetoed SB 1047, stating in part: “By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.”
PRIOR VOTES:
Senate Governmental Organization Committee (Ayes 13, Noes 0)
**************
Naomi Nix, Cat Zakrzewski & Gerrit De Vynck, Silicon Valley is pricing academics out of AI research (March 10, 2024) The Washington Post, https://www.washingtonpost.com/technology/2024/03/10/big-tech-companies-ai-research/ [as of Apr. 2, 2025]. ↑
SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION
Senator Steve Padilla
Chair
2025 - 2026 Regular
Bill No: SB Hearing Date: 3/25/2025
Author: | Wiener | ||
Version: | Amended | ||
Urgency: | Fiscal: | ||
Consultant: | |||
SUBJECT:
DIGEST: This bill establishes, within the Government Operations Agency (GovOps), a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute,” as specified. Additionally, this bill enhances whistleblower protections for employees involved with covered artificial intelligence (AI) models and covered model derivatives, as specified.
ANALYSIS:
Existing law:
- Establishes the California Department of Technology (CDT), within GovOps, and requires the department to conduct a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency, as specified.
- The Generative Artificial Intelligence (GenAI) Accountability Act requires CDT, under the guidance of GovOps, the Office of Data and Innovation (ODI), and the Department of Human Resources (CalHR), to update, as needed, the State of California Benefits and Risks of GenAI Intelligence Report.
- Defines “artificial intelligence” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- Prohibits employers and their agents from making, adopting, or enforcing a rule, regulation, or policy preventing an employee from disclosing information to certain entities or from providing information to, or testifying before, any public body conducting an investigation, hearing, or inquiry if the employee has reasonable cause to believe that the information discloses a violation of law, as specified, and prohibits retaliation against an employee for, among other things, exercising these rights.
- Provides that any report required or requested by law to be submitted by a state or local agency to a committee of the Legislature or the Members of either house of the Legislature generally, to instead be submitted as a printed copy to the Secretary of the Senate, as an electronic copy to the Chief Clerk of the Assembly, and as an electronic or printed copy to the Legislative Counsel, as specified.
This bill:
- Establishes a consortium, within GovOps, and requires the consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute” that advances the development and deployment of AI that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
- Fostering research and innovation that benefits the public.
- Enabling equitable innovation by expanding access to computational resources.
- Requires the consortium to make reasonable efforts to ensure that CalCompute is established within the University of California (UC) to the extent possible.
- Requires CalCompute to include, but not be limited to, all of the following:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate the use of CalCompute.
- Requires the consortium to operate in accordance with all relevant labor and workforce laws and standards.
- Requires GovOps, on or before January 1, 2027, to submit a report from the consortium to the Legislature with the framework developed pursuant to the above provisions for the creation and operation of CalCompute.
- Requires the report to include all of the following elements:
- A landscape analysis of California’s current public, private, and nonprofit cloud computing platform infrastructure.
- An analysis of the cost to the state to build and maintain CalCompute and recommendations for potential funding sources.
- Recommendations for the governance structure and ongoing operation of CalCompute.
- Recommendations for the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
- An analysis of the state’s technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
- A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements above.
- Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
- Requires the consortium to, consistent with state constitutional law, consist of 14 members selected from among all of the following:
- Representatives of the UC and other public and private academic research institutions and national laboratories.
- Representatives of impacted workforce labor organizations.
- Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
- Experts in technology and AI to provide technical assistance.
- Personnel from other relevant departments and agencies, as necessary.
- Requires the Secretary of GovOps to select eight members of the consortium, and the President pro Tempore of the Senate and the Speaker of the Assembly to each select three members.
- Specifies that if CalCompute is established within the UC, the UC may receive private donations for the purpose of implementing CalCompute.
- Provides that the above provisions shall become operative only upon an appropriation in a budget act, or other measure, as specified.
- Prohibits a developer, as defined, from making, adopting, or enforcing a rule, regulation, or policy that prevents an employee from disclosing, or from retaliating against an employee for disclosing, information to the Attorney General (AG), federal authorities, or another employee who has authority to investigate, discover, or correct the reported issue, as specified.
- Authorizes an employee, as defined, to use the whistleblower hotline maintained by the AG to make reports described above.
- Requires a developer to provide a clear notice to all employees of their rights and responsibilities under this bill, as specified.
- Requires a developer to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer’s activities present a critical risk, as specified.
- Requires the disclosures and responses of the process required by this bill to be shared with officers and directors of the developer at least once each quarter, as specified.
- Authorizes the court to award reasonable attorney’s fees to a plaintiff who brings a successful action for a violation of this section.
- Provides that in a civil action brought pursuant to this bill, the developer shall have the burden of proof to demonstrate by clear and convincing evidence that the alleged action would have occurred for legitimate, independent reasons even if the employee had not engaged in activities protected by this bill, as specified.
- Provides that in a civil action or administrative proceeding brought pursuant to this bill, an employee may petition the superior court in any county wherein the violation in question is alleged to have occurred, or wherein the person resides or transacts business, for appropriate temporary or preliminary injunctive relief.
- Provides, that upon the filing of the petition for injunctive relief, the petitioner shall cause notice thereof to be served upon the person, and thereupon the court shall have jurisdiction to grant temporary injunctive relief as the court deems just and proper.
- Provides that in addition to any harm resulting directly from a violation of this bill, the court shall consider the chilling effect on other employees asserting their rights under this bill in determining whether temporary injunctive relief is just and proper.
- Requires appropriate injunctive relief be issued on a showing that reasonable cause exists to believe a violation has occurred.
- Requires an order authorizing temporary injunctive relief to remain in effect until an administrative or judicial determination or citation has been issued, or until the completion of a review, as specified, whichever is longer, or at a certain time set by the court. Thereafter, a preliminary or permanent injunction may be issued if it is shown to be just and proper. Any temporary injunctive relief shall not prohibit a developer from disciplining or terminating an employee for conduct that is unrelated to the claim of the retaliation.
- Provides that injunctive relief granted pursuant to this bill shall not be stayed pending appeal, as specified.
- Includes relevant definitions of key words and terms, as specified.
Background
Author Statement. According to the author’s office, “the greatest innovations happen when our brightest minds have the resources they need and the freedom to speak their minds. California’s leadership on AI is more critical than ever as the new federal Administration proceeds with shredding the guardrails meant to keep Americans safe from the known and foreseeable risks that advanced AI systems present.”
Further, “Senate Bill 53 recognizes that we need a multipronged approach to encourage responsible AI innovation. It does so by: 1. Ensuring employees of frontier model labs are not retaliated against for reporting their belief that a developer’s activities pose a critical risk, as defined, or has made false or misleading statements about its management of critical risk; and, 2. Establishing a process to create a public cloud-computing cluster (CalCompute) that will conduct research into the safe and secure deployment of large-scale AI models.”
Cluster Computing. Cluster computing is a type of computing where multiple computers are connected to work together as a single system. The term “cluster” refers to the network of linked computer systems programmed performing the same task. Computing clusters typically consist of servers, workstations, and personal computers that communicate over a local area network or a wide area network. Cluster computing is a kind of distributed computing, a type of computing that links computers together on a network to perform a computational task, increase computational power and function as a single computer. Each computer, or “node,” in a cluster network has an operating system and a central processing unit core that handles the tasks required for the software to run properly.
Because of its high performance and high availability, cluster computing has many applications, including cloud computing, AI, scientific research, and big data analytics. Cloud computing is the on-demand access of computing resources – physical or virtual servers, data storage, networking capabilities, application development tools, software, AI-powered analytic platforms and more – over the internet with pay-per-use pricing. When you use cloud service, you’re accessing remote servers, powerful mainframe computers housed in large data centers, through the internet. The cloud computing model gives users greater flexibility and scalability compared to traditional on-premises infrastructure.
Examples include Amazon Web Services (AWS), which provides numerous clusters through services like Elastic Compute Cloud (EC2) and AWS Lambda. These cloud clusters power everything from simple web applications to complex AI training tasks. Google Cloud Platform (GCP) supports services like Google Compute Engine and Kubernetes Engine to simplify the deployment, management, and scaling of containerized applications, offering a scalable and automated environment for running containerized workloads. Microsoft Azure offers clusters through its Virtual Machines and Azure Kubernetes Service providing extensive AI and machine learning services integrated within its cloud environment. IBM Cloud and Oracle Cloud offer scalable computing clusters that support high-performance computing, big data applications, and AI workloads.
CalCompute. This bill requires GovOps to establish a consortium to develop a framework for the creation of a public cloud computing cluster to be known as “CalCompute.” By January 1, 2027, GovOps must submit to the Legislature the consortium’s report with the framework. This bill requires the framework for CalCompute to advance the development and deployment of AI that is safe, ethical, equitable, and sustainable by fostering research and innovation that benefits the public and enabling equitable innovation by expanding access to computational resources.
The consortium will be consist of eight members selected by the Secretary of GovOps, and three each by the President pro Tempore of the Senate and the Speaker of the Assembly. Specifically, this bill requires the 14 members be selected among representatives of the UC and other public and private academic research institutions and national laboratories; representatives of impacted workforce labor organizations; representatives of stakeholder groups with relevant expertise and experience; experts in technology and AI to provide technical assistance; and personnel from other relevant departments and agencies, as necessary.
This bill does not establish CalCompute, however, it does specify that if CalCompute is eventually established within the UC, the UC may receive private donations for the purposes of implementing CalCompute.
Supporters argue that a public CalCompute is a step towards democratizing access to advanced computational resources currently out of reach for many academics and startups in California. CalCompute can potentially lower the barriers to entry for innovative research and development in AI, and a shared resource can foster cross-sector collaboration, driving economic growth by leveling the playing field for smaller players that might otherwise be outpaced by large private firms.
Frontier Models and Safety. Frontier AI models are the cutting-edge class of “foundation” or general-purpose AI systems. These models have the ability to perform a wide variety of tasks – from natural language understanding to code generation and image creation – and are trained with incredible amounts of compute, data, energy, and water. These models are distinguished by their breadth of functionality and the scale of resources required to develop them (often costing hundreds of millions or billions of dollars). Their ability to produce powerful new capabilities potentially pose severe risks to safety or security if misused. Model derivatives are any downstream version of a frontier model created by modifying or adapting the original model’s weights, architecture, or outputs for specialized purposes and applications.
Frontier models, if misused, can give rise to serious harm such as launching large-scale cyberattacks that disrupt critical infrastructure and civil society – including power grids, communication networks, and financial systems – resulting in widespread economic and social chaos and uncertainty. These models can also be utilized to create autonomous systems capable of inflicting physical harm, for example, through the development of lethal autonomous weapons, the manipulation of transportation systems or dangerous recommendations from “therapy” chatbots. Additionally, and possibly most acutely, these models can be used to facilitate sophisticated disinformation campaigns that undermine public trust in democratic institutions destabilizing the political process. Other risks include privacy violations where sensitive or personal data might be exposed or misused.
Examples of frontier models include OpenAi’s ChatGPT, Google’s Gemini, Anthropic’s Claude, Meta’s Llama, X-Ai’s Grok, and Hangzhou’s Deep Seek. These models represent the frontier in natural language processing, having been trained on vast datasets and massive computational resources, enabling them to perform a wide variety of autonomous tasks.
Critical Risk and AI. This bill strengthens whistleblower protections for employees of frontier AI laboratory companies whose activities developing frontier AI pose a critical risk, as defined. These protections apply to companies that have trained at least one foundation model with a quantity of computational power that costs at least $100 million. Specifically, this bill prevents developers of these frontier models from making, adopting, or enforcing any rule, regulation, or policy that prevents an employee from disclosing, or retaliates against an employee for disclosing, information to the AG, federal authorities, or another employee, as specified, if the employee has reasonable cause to believe that the information discloses that the developer’s activities pose a critical risk or the developer has made false or misleading statements about its management of critical risk.
Should this bill advance in this committee, it will be referred to the Senate Committee on the Judiciary which has appropriate jurisdiction to further discuss the whistleblower provisions of this bill.
Back to the Future. SB 1047 (Wiener, 2024) of last year would have, among other things, required developers of large AI models, and those providing the computing power to train such models, to put certain safeguards and policies in place to prevent catastrophic harm, as defined. Similarly to this bill, SB 1047 would have also established a consortium in GovOps to develop a framework for the creation of CalCompute and included enhanced whistleblower protections for employees that have reasonable cause to believe a developer is out of compliance with certain requirements for that covered AI model, as specified.
SB 1047 was vetoed by Governor Newsom who wrote, in part:
Let me be clear – I agree with the author – we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
To those who say there’s no problem here to solve, or that California does not have a role in regulating potential national security implications of this technology, I disagree. A California-only approach may well be warranted – especially absent federal action by Congress – but it must be based on empirical evidence and science. The U.S. AI Safety Institute, under the National Institute of Science and Technology, is developing guidance on national security risks, informed by evidence-based approaches, to guard against demonstrable risks to public safety. Under and Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of potential threats and vulnerabilities to California’s critical infrastructure using AI. These are just a few examples of the many endeavors underway, led by experts, to inform policymakers on AI risk management practices that are rooted in science and fact. And endeavors like these have led to the introduction of over a dozen bills regulating specific, known risks posed by AI, that I have signed in the last 30 days.
Suggested Amendments. The author has agreed to the following amendments:
Amendment #1: 11547.61(h) The members of the consortium shall serve without compensation, but shall be reimbursed for all necessary expenses actually incurred in the performance of their duties.
Amendment #2: 11547.61(i) The working group shall be dissolved upon submission of the report required by paragraph (1) of subdivision (f) to the Legislature.
Prior/Related Legislation
SB 579 (Padilla, 2025) requires the Secretary of GovOps to appoint a mental health and AI working group to evaluate certain issues and determine the role of AI in mental health settings, as specified. (Pending in the Senate Governmental Organization Committee)
AB 1064 (Bauer-Kahan, 2025) the Leading Ethical AI Development (LEAD) for Kids Act, among other things, establishes the LEAD for Kids Standards Board, in GovOps, to adopt regulations governing criteria for determining the level of estimated risk of a covered product based on an analysis that weighs the likelihood and severity of reasonably foreseeable adverse impacts against the anticipated benefits of the covered product and denominating the risk levels, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)
SB 893 (Padilla, 2023) would have required GovOps, the Governor’s Office of Business and Economic Development, the California Privacy Protection Agency, and CDT to establish the California AI Research Hub, as specified. (Held in the Assembly Appropriations Committee Suspense File)
SB 1047 (Wiener, 2024) would have required developers of powerful AI models, as defined, and those providing the computer power to train such models to put appropriate safeguards and polici4es in place to prevent critical harms, as defined. Would have established a state entity to oversee the development of these models and called for the creation of a consortium to develop a framework for a public cloud computing cluster. (Vetoed by Governor Newsom)
AB 2885 (Bauer-Kahan, Chapter 843, Statutes of 2024) defined “artificial intelligence” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights” and expresses the Legislature’s commitment to examining and implement those principles in its legislation and policies related to the use and deployment of automated systems.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
SUPPORT:
Secure AI Project (Co-source)
Economic Security California Action (Co-source)
Encode (Co-source)
AI Policy Tracker
Apart Research
Center for AI and Digital Policy
Center for Human-Compatible Artificial Intelligence, UC Berkeley
Compute Exchange
Nonlinear
Oakland Privacy
Redwood Research
Safe AI Future
OPPOSITION:
None received
ARGUMENTS IN SUPPORT: In support of the bill, the co-sources write that, “AI researchers and engineers will likely be the first to know if a company’s product poses a significant risk to public safety, but company contracts may prevent them from reporting concerns. In general, companies offer or require broad non-disclosure and non-disparagement agreements that are quite broad, often limiting nearly all disclosures that are not explicitly protected by law. These agreements in practice can prevent employees from sharing a good faith public safety concern with a government authority, which could chill employees’ disclosure rights when that is their best or only course of action.”
In addition to “whistleblower protections, SB 53 would also set in motion CalCompute, a public cloud computing cluster for use by academics and startups in California. Computational resources are essential for AI research, and CalCompute would make those resources more accessible to California’s top universities and startups, helping to catalyze additional research into beneficial applications of AI and supporting in particular smaller startups for a healthier innovation ecosystem. We support this groundbreaking effort, which would advance and democratize AI research in California.”
DUAL-REFERRAL: Senate Committee on Governmental Organization & Senate Committee on Judiciary
Discussed in Hearing
Assembly Floor
Senate Floor
Assembly Standing Committee on Privacy and Consumer Protection
Assembly Standing Committee on Privacy and Consumer Protection
Assembly Standing Committee on Judiciary
Senate Floor
Senate Standing Committee on Governmental Organization
Bill Author
Bill Co-Author(s):