SB 1047: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
- Session Year: 2023-2024
- House: Senate
- Latest Version Date: 2024-09-03
Current Status:
Failed
(2024-09-29: In Senate. Consideration of Governor's veto pending.)
Introduced
In Committee
First Chamber
In Committee
Second Chamber
Enacted
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require that a developer, before beginning to initially train a covered model, as defined, comply with various requirements, including implementing the capability to promptly enact a full shutdown, as defined, and implement a written and separate safety and security protocol, as specified. The bill would require a developer to retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years, including records and dates of any updates or revisions and would require a developer to grant to the Attorney General access to the unredacted safety and security protocol. The bill would prohibit a developer from using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or a covered model derivative available for commercial or public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative will cause or materially enable a critical harm, as defined. The bill would require a developer, beginning January 1, 2026, to annually retain a third-party auditor to perform an independent audit of compliance with those provisions, as prescribed. The bill would require the auditor to produce an audit report, as prescribed, and would require a developer to retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years. The bill would require a developer to grant to the Attorney General access to the unredacted auditors report upon request. The bill would exempt from disclosure under the California Public Records Act the safety and security protocol and the auditors report described above.
This bill would require a developer of a covered model to submit to the Attorney General a statement of compliance with these provisions, as specified. The bill would also require a developer of a covered model to report each artificial intelligence safety incident affecting the covered model or any covered model derivative controlled by the developer to the Attorney General, as prescribed.
This bill would require a person that operates a computing cluster, as defined, to implement written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
This bill would authorize the Attorney General to bring a civil action, as provided. The bill would also provide for whistleblower protections, including by prohibiting a developer of a covered model or a contractor or subcontractor of the developer from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
This bill would create the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the boards membership. The bill would require the Government Operations Agency to, on or before January 1, 2027, and annually thereafter, issue regulations to, among other things, update the definition of a covered model, as provided, and would require the regulations to be approved by the board before taking effect.
This bill would establish in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would, on or before January 1, 2026, require the Government Operations Agency to submit a report from the consortium to the Legislature with that framework. The bill would make those provisions operative only upon an appropriation in a budget act for its purposes.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require that a developer, before beginning to initially train a covered model, as defined, comply with various requirements, including implementing the capability to promptly enact a full shutdown, as defined, and implement a written and separate safety and security protocol, as specified. The bill would require a developer to retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years, including records and dates of any updates or revisions and would require a developer to grant to the Attorney General access to the unredacted safety and security protocol. The bill would prohibit a developer from using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or a covered model derivative available for commercial or public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative will cause or materially enable a critical harm, as defined. The bill would require a developer, beginning January 1, 2026, to annually retain a third-party auditor to perform an independent audit of compliance with those provisions, as prescribed. The bill would require the auditor to produce an audit report, as prescribed, and would require a developer to retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years. The bill would require a developer to grant to the Attorney General access to the unredacted auditors report upon request. The bill would exempt from disclosure under the California Public Records Act the safety and security protocol and the auditors report described above.
This bill would require a developer of a covered model to submit to the Attorney General a statement of compliance with these provisions, as specified. The bill would also require a developer of a covered model to report each artificial intelligence safety incident affecting the covered model, or any covered model derivative controlled by the developer to the Attorney General, as prescribed.
This bill would require a person that operates a computing cluster, as defined, to implement written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
This bill would authorize the Attorney General to bring a civil action, as provided. The bill would also provide for whistleblower protections, including by prohibiting a developer of a covered model or a contractor or subcontractor of the developer from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
This bill would create the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the boards membership. The bill would require the Government Operations Agency to, on or before January 1, 2027, and annually thereafter, issue regulations to, among other things, update the definition of a covered model, as provided, and would require the regulations to be approved by the board before taking effect.
This bill would establish in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would on or before January 1, 2026, require the Government Operations Agency to submit a report from the consortium to the Legislature with that framework. The bill would make those provisions operable only upon an appropriation in a budget act for its purposes.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require that a developer, before beginning to initially training train a covered model, as defined, comply with various requirements, including implementing the capability to promptly enact a full shutdown, as defined, and implement a written and separate safety and security protocol, as specified. The bill would require a developer to retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years, including records and dates of any updates or revisions and would require a developer to grant to the Attorney General access to the unredacted safety and security protocol. The bill would prohibit a developer from using a covered model commercially or publicly, or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or a covered model derivative available for commercial or public public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative can will cause or materially enable a critical harm, as defined. The bill would require a developer, beginning January 1, 2028, 2026, to annually retain a third-party auditor to perform an independent audit of compliance with the requirements of the bill, as provided. those provisions, as prescribed. The bill would require the auditor to produce an audit report, as prescribed, and would require a developer to retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus 5 years. The bill would require a developer to grant to the Attorney General access to the unredacted auditors report upon request. The bill would exempt from disclosure under the California Public Records Act the safety and security protocol and the auditors report described above.
This bill would require a developer of a covered model to submit to the Frontier Model Division, which the bill would create within the Government Operations Agency, a certification under penalty of perjury Attorney General a statement of compliance with these provisions, as specified. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer of a covered model to report each artificial intelligence safety incident affecting the covered model model, or any covered model derivative controlled by the developer, as specified, developer to the Frontier Model Division. Attorney General, as prescribed.
This bill would require a person that operates a computing cluster, as defined, to implement written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
This bill would specify unlawful acts under these provisions and authorize the Attorney General or the Labor Commissioner to bring a civil action, as provided. The bill would also provide for whistleblower protections, including by prohibiting a developer of a covered model or a contractor or subcontractor of the developer from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
This bill would create the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the boards membership. The bill would also create the Frontier Model Division within the Government Operations Agency and under the direct supervision of the board, and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would require the division Government Operations Agency to, on or before January 1, 2027, and annually thereafter, issue regulations to to, among other things, update the definition of a covered model, as provided. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature. provided, and would require the regulations to be approved by the board before taking effect.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.This bill would provide that no reimbursement is required by this act for a specified reason.This bill would establish in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by, among other things, fostering research and innovation that benefits the public, as prescribed. The bill would on or before January 1, 2026, require the Government Operations Agency to submit a report from the consortium to the Legislature with that framework. The bill would make those provisions operable only upon an appropriation in a budget act for its purposes.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require that a developer, before initially training a covered model, as defined, comply with various requirements, including implementing the capability to promptly enact a full shutdown, as defined, and implement a written and separate safety and security protocol, as specified. The bill would prohibit a developer from using a covered model commercially or publicly, or making a covered model or a covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm, as defined. The bill would require a developer, beginning January 1, 2028, to annually retain a third-party auditor to perform an independent audit of compliance with the requirements of the bill, as provided.
This bill would require a developer of a covered model to submit to the Frontier Model Division, which the bill would create within the Government Operations Agency, a certification under penalty of perjury of compliance with these provisions, as specified. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer of a covered model to report each artificial intelligence safety incident affecting the covered model or any covered model derivative controlled by the developer, as specified, to the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.This bill would specify unlawful acts under these provisions and authorize the Attorney General or the Labor Commissioner to bring a civil action, as provided. The bill would also provide for whistleblower protections, including prohibiting a developer of a covered model or a contractor or subcontractor of the developer from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
This bill would create the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the boards membership. The bill would also create the Frontier Model Division within the Government Operations Agency and under the direct supervision of the board, and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would require the division to, on or before January 1, 2027, and annually thereafter, issue regulations to update the definition of a covered model, as provided. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, authorize a developer of a covered model, as defined, to determine whether a covered model qualifies for a limited duty exemption before initiating training of that covered model, as specified. The bill would define limited duty exemption to mean a determination, made as specified, with respect to a covered model, that is not a derivative model, that a developer can provide reasonable assurance that the covered model does not have a hazardous capability, as defined, and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative initially training a covered model, as defined, comply with various requirements, including implementing the capability to promptly enact a full shutdown, as defined, of the covered model until that covered model is the subject of a limited duty exemption. and implement a written and separate safety and security protocol, as specified. The bill would prohibit a developer from using a covered model commercially or publicly, or making a covered model or a covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm, as defined.
This bill would require a developer of a nonderivative covered model that is not the subject of a limited duty exemption to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual Government Operations Agency, a certification under penalty of perjury of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. provisions, as specified. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer of a nonderivative covered model to report each artificial intelligence safety incident affecting that the covered model and or any covered model derivative version of that covered model within the custody, control, or possession of the controlled by the developer, as specified, to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy train a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would create the Board of Frontier Models within the Government Operations Agency, independent of the Department of Technology, and provide for the boards membership. The bill would also create the Frontier Model Division within the Department of Technology the Government Operations Agency and under the direct supervision of the board, and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would require the division to, on or before January 1, 2027, and annually thereafter, issue regulations to update the definition of a covered model, as provided. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, authorize a developer of a covered model, as defined, to determine whether a covered model qualifies for a limited duty exemption before initiating training of that covered model, as specified. The bill would define limited duty exemption to mean a determination, made as specified, with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility provide reasonable assurance that the covered model has does not have a hazardous capability, as defined, or may and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown shutdown, as defined, of the covered model until that covered model is the subject of a limited duty exemption.
This bill would require a developer of a nonderivative covered model that is not the subject of a limited duty exemption to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification under penalty of perjury of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer of a nonderivative covered model to report each artificial intelligence safety incident affecting a that covered model and any derivative version of that covered model within the custody, control, or possession of the developer, as specified, to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, authorize a developer of a covered model, as defined, to determine whether a covered model qualifies for a limited duty exemption before initiating training of that covered model, as specified. The bill would define limited duty exemption to mean a determination, made as specified, with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility that the covered model has a hazardous capability, as defined, or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown of the covered model until that covered model is the subject of a limited duty exemption.
This bill would require a developer of a nonderivative covered model that is not the subject of a limited duty exemption to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification under penalty of perjury of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer to report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, authorize a developer of a covered model, as defined, to determine whether a covered model qualifies for a limited duty exemption before initiating training of that covered model, as specified. The bill would define limited duty exemption to mean a determination, made as specified, with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility that the covered model has a hazardous capability, as defined, or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown of the covered model until that covered model is the subject of a limited duty exemption.
This bill would require a developer of a nonderivative covered model that is not the subject of a limited duty exemption to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification under penalty of perjury of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer to report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require authorize a developer of a covered model, as defined, to determine whether it can make a positive safety determination with respect to a covered model qualifies for a limited duty exemption before initiating training of that covered model, as specified. The bill would define positive safety determination limited duty exemption to mean a determination determination, made as specified, with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility that the covered model has a hazardous capability, as defined, or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown of the covered model until that covered model is the subject of a positive safety determination. limited duty exemption.
This bill would require a developer of a nonderivative covered model that is not the subject of a positive safety determination limited duty exemption to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification under penalty of perjury of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer to report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act to, among other things, require a developer of a covered model, as defined, to determine whether it can make a positive safety determination with respect to a covered model before initiating training of that covered model, as specified. The bill would define positive safety determination to mean a determination with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility that the covered model has a hazardous capability, as defined, or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown of the covered model until that covered model is the subject of a positive safety determination.
This bill would require a developer of a nonderivative covered model that is not the subject of a positive safety determination to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer to report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Systems Models Act to, among other things, require a developer of a covered model, as defined, to determine whether it can make a positive safety determination with respect to a covered model before initiating training of that covered model, as specified. The bill would define positive safety determination to mean a determination with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility that the covered model has a hazardous capability, as defined, or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown of the covered model until that covered model is the subject of a positive safety determination.
This bill would require a developer of a nonderivative covered model that is not the subject of a positive safety determination to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer to report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model. The bill
This bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
Existing law creates the Department of Technology within the Government Operations Agency and requires the department to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
This bill would enact the Safe and Secure Innovation for Frontier Artificial Intelligence Systems Act to, among other things, require a developer of a covered model, as defined, to determine whether it can make a positive safety determination with respect to a covered model before initiating training of that covered model, as specified. The bill would define positive safety determination to mean a determination with respect to a covered model, that is not a derivative model, that a developer can reasonably exclude the possibility that the covered model has a hazardous capability, as defined, or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
This bill would require that a developer, before initiating training of a nonderivative covered model, comply with various requirements, including implementing the capability to promptly enact a full shutdown of the covered model until that covered model is the subject of a positive safety determination.
This bill would require a developer of a nonderivative covered model that is not the subject of a positive safety determination to submit to the Frontier Model Division, which the bill would create within the Department of Technology, an annual certification of compliance with these provisions signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program. The bill would also require a developer to report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
This bill would require a person that operates a computing cluster, as defined, to implement appropriate written policies and procedures to do certain things when a customer utilizes compute resources that would be sufficient to train a covered model, including assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model. The bill would punish a violation of these provisions with a civil penalty, as prescribed, to be recovered by the Attorney General.
This bill would also create the Frontier Model Division within the Department of Technology and would require the division to, among other things, review annual certification reports from developers received pursuant to these provisions and publicly release summarized findings based on those reports. The bill would authorize the division to assess related fees and would require deposit of the fees into the Frontier Model Division Programs Fund, which the bill would create. The bill would make moneys in the fund available for the purpose of these provisions only upon appropriation by the Legislature.
This bill would also require the Department of Technology to commission consultants, as prescribed, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, among other things, a fully owned and hosted cloud platform.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Artificial intelligence safety incident means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
(1)A covered model or covered model derivative autonomously engaging in behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model or covered model derivative.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model or covered model derivative.
(4)Unauthorized use of a covered model or covered model derivative to cause or materially enable critical harm.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)(1)Covered model means either of the following:
(A)Before January 1, 2027, covered model means either of the following:
(i)An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market prices of cloud compute at the start of training as reasonably assessed by the developer.
(ii)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than three times 10^25 integer or floating-point operations, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(B)(i)Except as provided in clause (ii), on and after January 1, 2027, covered model means any of the following:
(I)An artificial intelligence model trained using a quantity of computing power determined by the Government Operations Agency pursuant to Section 11547.6 of the Government Code, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
(II)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Government Operations Agency, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(ii)If the Government Operations Agency does not adopt a regulation governing subclauses (I) and (II) of clause (i) before January 1, 2027, the definition of covered model in subparagraph (A) shall be operative until the regulation is adopted.
(2)On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(f)Covered model derivative means any of the following:
(1)An unmodified copy of a covered model.
(2)A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
(3)(A)(i)Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding three times 10^25 integer or floating point operations, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(ii)On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Government Operations Agency, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(B)If the Government Operations Agency does not adopt a regulation governing clause (ii) of subparagraph (A) by January 1, 2027, the quantity of computing power specified in clause (i) of subparagraph (A) shall continue to apply until the regulation is adopted.
(4)A copy of a covered model that has been combined with other software.
(g)(1)Critical harm means any of the following harms caused or materially enabled by a covered model or covered model derivative:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model conducting, or providing precise instructions for conducting, a cyberattack or series of cyberattacks on critical infrastructure.
(C)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that does both of the following:
(i)Acts with limited human oversight, intervention, or supervision.
(ii)Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
(D)Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2)Critical harm does not include any of the following:
(A)Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.
(B)Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other softwares ability to cause or materially enable the harm.
(C)Harms that are not caused or materially enabled by the developers creation, storage, use, or release of a covered model or covered model derivative.
(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)Developer means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power and cost, or by fine-tuning an existing covered model or covered model derivative using a quantity of computing power and cost greater than the amount specified in subdivision (e).
(j)Fine-tuning means adjusting the model weights of a trained covered model or covered model derivative by exposing it to additional data.
(k)Full shutdown means the cessation of operation of all of the following:
(1)The training of a covered model.
(2)A covered model controlled by a developer.
(3)All covered model derivatives controlled by a developer.
(l)Model weight means a numerical parameter in an artificial intelligence model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(m)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(n)Post-training modification means modifying the capabilities of a covered model or covered model derivative by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(o)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models and covered model derivatives across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of covered model derivatives.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model and covered model derivatives.
22603.(a)Before beginning to initially train a covered model, the developer shall do all of the following:
(1)Implement reasonable administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)(A)Implement the capability to promptly enact a full shutdown.
(B)When enacting a full shutdown, the developer shall take into account, as appropriate, the risk that a shutdown of the covered model, or particular covered model derivatives, could cause disruptions to critical infrastructure.
(3)Implement a written and separate safety and security protocol that does all of the following:
(A)Specifies protections and procedures that, if successfully implemented, would successfully comply with the developers duty to take reasonable care to avoid producing a covered model or covered model derivative that poses an unreasonable risk of causing or materially enabling a critical harm.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies a testing procedure, which takes safeguards into account as appropriate, that takes reasonable care to evaluate if both of the following are true:
(i)A covered model poses an unreasonable risk of causing or enabling a critical harm.
(ii)Covered model derivatives pose an unreasonable risk of causing or enabling a critical harm.
(D)Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
(E)Describes in detail how the testing procedure addresses the possibility that a covered model or covered model derivative can be used to make post-training modifications or create another covered model in a manner that may cause or materially enable a critical harm.
(F)Describes in detail how the developer will fulfill their obligations under this chapter.
(G)Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.
(H)Describes in detail the conditions under which a developer would enact a full shutdown.
(I)Describes in detail the procedure by which the safety and security protocol may be modified.
(4)Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, or any covered model derivatives controlled by the developer, monitoring and reporting on implementation.
(5)Retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years, including records and dates of any updates or revisions.
(6)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(7)(A)(i)Conspicuously publish a copy of the redacted safety and security protocol and transmit a copy of the redacted safety and security protocol to the Attorney General.
(ii)A redaction in the safety and security protocol may be made only if the redaction is reasonably necessary to protect any of the following:
(I)Public safety.
(II)Trade secrets, as defined in Section 3426.1 of the Civil Code.
(III)Confidential information pursuant to state and federal law.
(B)The developer shall grant to the Attorney General access to the unredacted safety and security protocol upon request.
(C)A safety and security protocol disclosed to the Attorney General pursuant to this paragraph is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(D)If the safety and security protocol is materially modified, conspicuously publish and transmit to the Attorney General an updated redacted copy within 30 days of the modification.
(8)Take reasonable care to implement other appropriate measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or materially enabling critical harms.
(b)Before using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or before making a covered model or covered model derivative available for commercial or public, or foreseeably public, use, the developer of a covered model shall do all of the following:
(1)Assess whether the covered model is reasonably capable of causing or materially enabling a critical harm.
(2)Record, as and when reasonably possible, and retain for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years information on the specific tests and test results used in the assessment pursuant to paragraph (1) that provides sufficient detail for third parties to replicate the testing procedure.
(3)Take reasonable care to implement appropriate safeguards to prevent the covered model and covered model derivatives from causing or materially enabling a critical harm.
(4)Take reasonable care to ensure, to the extent reasonably possible, that the covered models actions and the actions of covered model derivatives, as well as critical harms resulting from their actions, can be accurately and reliably attributed to them.
(c)A developer shall not use a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or make a covered model or a covered model derivative available for commercial or public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative will cause or materially enable a critical harm.
(d)A developer of a covered model shall annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section.
(e)(1)Beginning January 1, 2026, a developer of a covered model shall annually retain a third-party auditor that conducts audits consistent with best practices for auditors to perform an independent audit of compliance with the requirements of this section.
(2)An auditor shall conduct audits consistent with regulations issued by the Government Operations Agency pursuant to subdivision (d) of Section 11547.6 of the Government Code.
(3)The auditor shall be granted access to unredacted materials as necessary to comply with the auditors obligations under this subdivision.
(4)The auditor shall produce an audit report including all of the following:
(A)A detailed assessment of the developers steps to comply with the requirements of this section.
(B)If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
(C)A detailed assessment of the developers internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
(D)The signature of the lead auditor certifying the results of the auditor.
(5)The developer shall retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years.
(6)(A)(i)The developer shall conspicuously publish a redacted copy of the auditors report and transmit to the Attorney General a copy of the redacted auditors report.
(ii)A redaction in the auditors report may be made only if the redaction is reasonably necessary to protect any of the following:
(I)Public safety.
(II)Trade secrets, as defined in Section 3426.1 of the Civil Code.
(III)Confidential information pursuant to state and federal law.
(B)The developer shall grant to the Attorney General access to the unredacted auditors report upon request.
(C)An auditors report disclosed to the Attorney General pursuant to this paragraph is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(7)An auditor shall not knowingly make a material misrepresentation in the auditors report.
(f)(1)(A)A developer of a covered model shall annually submit to the Attorney General a statement of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, that meets the requirements of paragraph (2).
(B)This paragraph applies if the covered model or any covered model derivatives controlled by the developer remain in commercial or public use or remain available for commercial or public use.
(2)In a statement submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)An assessment of the nature and magnitude of critical harms that the covered model or covered model derivatives may reasonably cause or materially enable and the outcome of the assessment required by paragraph (1) of subdivision (b).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent the covered model or covered model derivatives from causing or materially enabling critical harms.
(C)A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the statement and the contact information of any third parties relied upon to validate compliance.
(g)A developer of a covered model shall report each artificial intelligence safety incident affecting the covered model, or any covered model derivatives controlled by the developer, to the Attorney General within 72 hours of the developer learning of the artificial intelligence safety incident or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)A developer shall submit to the Attorney General a statement described by subdivision (f) no more than 30 days after using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or covered model derivative available for commercial or public, or foreseeably public, use for the first time.
(2)This subdivision does not apply with respect to a covered model derivative if the developer submitted a statement described by subdivision (f) for the applicable covered model from which the covered model derivative is derived.
(i)In fulfilling its obligations under this chapter, a developer shall consider industry best practices and applicable guidance from the U.S. Artificial Intelligence Safety Institute, National Institute of Standards and Technology, the Government Operations Agency, and other reputable standard-setting organizations.
(j)(1)This section shall not apply to products or services to the extent that the requirements would strictly conflict with the terms of a contract with a federal government entity and a developer of a covered model.
(2)This section applies to the development, use, or commercial or public release of a covered model or covered model derivative for any use that is not the subject of a contract with a federal government entity, even if that covered model or covered model derivative has already been developed, trained, or used by a federal government entity.
22604.(a)A person that operates a computing cluster shall implement written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(1)Obtain the prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(A)The identity of the prospective customer.
(B)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(C)The email address and telephonic contact information used to verify the prospective customers identity.
(2)Assess whether the prospective customer intends to utilize the computing cluster to train a covered model.
(3)If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to paragraph (1) and conduct the assessment required pursuant to paragraph (2) prior to each utilization.
(4)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(5)Maintain for seven years and provide to the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(6)Implement the capability to promptly enact a full shutdown of any resources being used to train or operate models under the customers control.
(b)A person that operates a computing cluster shall consider industry best practices and applicable guidance from the U.S. Artificial Intelligence Safety Institute, National Institute of Standards and Technology, and other reputable standard-setting organizations.
(c)In complying with the requirements of this section, a person that operates a computing cluster may impose reasonable requirements on customers to prevent the collection or retention of personal information that the person that operates a computing cluster would not otherwise collect or retain, including a requirement that a corporate customer submit corporate contact information rather than information that would identify a specific individual.
22606.(a)The Attorney General may bring a civil action for a violation of this chapter and to recover all of the following:
(1)For a violation that causes death or bodily harm to another human, harm to property, theft or misappropriation of property, or that constitutes an imminent risk or threat to public safety that occurs on or after January 1, 2026, a civil penalty in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(2)For a violation of Section 22607 that would constitute a violation of the Labor Code, a civil penalty specified in subdivision (f) of Section 1102.5 of the Labor Code.
(3)For a person that operates a computing cluster for a violation of Section 22604, for an auditor for a violation of paragraph (6) of subdivision (e) of Section 22603, or for an auditor who intentionally or with reckless disregard violates a provision of subdivision (e) of Section 22603 other than paragraph (6) or regulations issued by the Government Operations Agency pursuant to Section 11547.6 of the Government Code, a civil penalty in an amount not exceeding fifty thousand dollars ($50,000) for a first violation of Section 22604, not exceeding one hundred thousand dollars ($100,000) for any subsequent violation, and not exceeding ten million dollars ($10,000,000) in the aggregate for related violations.
(4)Injunctive or declaratory relief.
(5)(A)Monetary damages.
(B)Punitive damages pursuant to subdivision (a) of Section 3294 of the Civil Code.
(6)Attorneys fees and costs.
(7)Any other relief that the court deems appropriate.
(b)In determining whether the developer exercised reasonable care as required in Section 22603, all of the following considerations are relevant but not conclusive:
(1)The quality of a developers safety and security protocol.
(2)The extent to which the developer faithfully implemented and followed its safety and security protocol.
(3)Whether, in quality and implementation, the developers safety and security protocol was inferior, comparable, or superior to those of developers of comparably powerful models.
(4)The quality and rigor of the developers investigation, documentation, evaluation, and management of risks of critical harm posed by its model.
(c)(1)A provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of this chapter, or to shift that liability to any person or entity in exchange for their use or access of, or right to use or access, a developers products or services, including by means of a contract of adhesion, is void as a matter of public policy.
(2)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section to the maximum extent allowed by law if the court concludes that both of the following are true:
(A)The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
(B)As the result of the steps described in subparagraph (A), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties, damages, or injunctive relief under this section.
(d)Penalties collected pursuant to this section by the Attorney General shall be deposited into the Public Rights Law Enforcement Special Fund established pursuant to Section 12530 of the Government Code.
(e)This section does not limit the application of other laws.
22607.(a)A developer of a covered model or a contractor or subcontractor of the developer shall not do any of the following:
(1)Prevent an employee from disclosing information to the Attorney General or the Labor Commissioner, including through terms and conditions of employment or seeking to enforce terms and conditions of employment if the employee has reasonable cause to believe the information indicates either of the following:
(A)The developer is out of compliance with the requirements of Section 22603.
(B)An artificial intelligence model, including a model that is not a covered model or a covered model derivative, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
(2)Retaliate against an employee for disclosing information to the Attorney General or the Labor Commissioner pursuant to paragraph (1).
(3)Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 (commencing with Section 16600) of Division 7 or any other provision of state law.
(b)An employee harmed by a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
(c)(1)The Attorney General or Labor Commissioner may publicly release or provide to the Governor any complaint, or a summary of that complaint, pursuant to this section if the Attorney General or the Labor Commissioner concludes that doing so will serve the public interest.
(2)If the Attorney General or the Labor Commissioner publicly releases a complaint, or a summary of a complaint, pursuant to paragraph (1), the Attorney General or the Labor Commissioner shall redact from the complaint any information that is confidential or otherwise exempt from public disclosure pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code) and any information that the Attorney General or the Labor Commissioner determines would likely pose an unreasonable risk to public safety if it were disclosed to the public.
(d)A developer shall provide a clear notice to all employees working on covered models and covered model derivatives of their rights and responsibilities under this section, including the right of employees of contractors and subcontractors to use the developers internal process for making protected disclosures pursuant to subdivision (e). A developer is presumed to be in compliance with the requirements of this subdivision if the developer does either of the following:
(1)At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice, and ensure that employees who work remotely periodically receive an equivalent notice.
(2)No less frequently than once every year, provides written notice to all employees of their rights and responsibilities under this chapter and ensures that the notice is received and acknowledged by all of those employees.
(e)(1)(A)A developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer has violated any provision of Section 22603 or any other law, or has made false or materially misleading statements related to its safety and security protocol, or failed to disclose known risks to employees, including, at a minimum, a monthly update to the person who made the disclosure regarding the status of the developers investigation of the disclosure and the actions taken by the developer in response to the disclosure.
(B)The process required by this paragraph shall apply to employees of the developers contractors and subcontractors working on covered models and covered model derivatives and allow those employees to disclose the same information to the developer that an employee of the developer may disclose and provide the same anonymity and protections against retaliation to the employees of the contractor or subcontractor that apply to disclosures by employees of the developer.
(2)The disclosures and responses of the process required by this subdivision shall be maintained for a minimum of seven years from the date when the disclosure or response is created. Each disclosure and response shall be shared with officers and directors of the developer whose acts or omissions are not implicated by the disclosure or response no less frequently than once per quarter. In the case of a report or disclosure regarding alleged misconduct by a contractor or subcontractor, the developer shall notify the officers and directors of the contractor or subcontractor whose acts or omissions are not implicated by the disclosure or response about the status of their investigation no less frequently than once per quarter.
(f)This section does not limit protections provided to employees by Section 1102.5 of the Labor Code, Section 12964.5 of the Government Code, or other law.
(g)As used in this section:
(1)Employee has the same meaning as defined in Section 1132.4 of the Labor Code and includes both of the following:
(A)Contractors or subcontractors and unpaid advisors involved with assessing, managing, or addressing the risk of critical harm from covered models and covered model derivatives.
(B)Corporate officers.
(2)Contractor or subcontractor has the same meaning as in Section 1777.1 of the Labor Code.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
22609.This chapter does not apply to the extent that it is preempted by federal law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section, critical harm has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)There is hereby established the Board of Frontier Models. The board shall be housed in the Government Operations Agency and shall be independent of the Department of Technology. The Governor may appoint an executive officer of the board, subject to Senate confirmation, who shall hold the office at the pleasure of the Governor. The executive officer shall be the administrative head of the board and shall exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged.
(c)(1)Commencing January 1, 2026, the Board of Frontier Models shall be composed of nine members, as follows:
(A)A member of the open-source community appointed by the Governor and subject to Senate confirmation.
(B)A member of the artificial intelligence industry appointed by the Governor and subject to Senate confirmation.
(C)An expert in chemical, biological, radiological, or nuclear weapons appointed by the Governor and subject to Senate confirmation.
(D)An expert in artificial intelligence safety appointed by the Governor and subject to Senate confirmation.
(E)An expert in cybersecurity of critical infrastructure appointed by the Governor and subject to Senate confirmation.
(F)Two members who are academics with expertise in artificial intelligence appointed by the Speaker of the Assembly.
(G)Two members appointed by the Senate Rules Committee.
(2)A member of the Board of Frontier Models shall meet all of the following criteria:
(A)A member shall be free of direct and indirect external influence and shall not seek or take instructions from another.
(B)A member shall not take an action or engage in an occupation, whether gainful or not, that is incompatible with the members duties.
(C)A member shall not, either at the time of the members appointment or during the members term, have a financial interest in an entity that is subject to regulation by the board.
(3)A member of the board shall serve at the pleasure of the members appointing authority but shall serve for no longer than eight consecutive years.
(d)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue regulations to update both of the following thresholds in the definition of a covered model to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose significant risk of causing or materially enabling critical harms.
(2)The updated definition shall contain both of the following:
(A)The initial compute threshold that an artificial intelligence model shall exceed to be considered a covered model.
(B)The fine-tuning compute threshold that an artificial intelligence model shall meet to be considered a covered model.
(3)In developing regulations pursuant to this subdivision, the Government Operations Agency shall take into account all of the following:
(A)The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or materially enable a critical harm.
(B)Similar thresholds used in federal law, guidance, or regulations for the management of artificial intelligence models with reasonable risks of causing or enabling critical harms.
(C)Input from stakeholders, including academics, industry, the open-source community, and government entities.
(e)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue regulations to establish binding auditing requirements applicable to audits conducted pursuant to subdivision (e) of Section 22603 of the Business and Professions Code to ensure the integrity, independence, efficiency, and effectiveness of the auditing process. In developing regulations pursuant to this subdivision, the Government Operations Agency shall take into account both of the following:
(A)Relevant standards or requirements imposed under federal or state law or through self-regulatory or standards-setting bodies.
(B)Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
(2)Any regulations issued pursuant to paragraph (1) shall, at a minimum, be consistent with guidance issued by the U.S. Artificial Intelligence Safety Institute and the National Institute of Standards and Technology.
(f)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue guidance for preventing unreasonable risks of covered models and covered model derivatives causing or materially enabling critical harms, including, but not limited to, more specific components of, or requirements under, the duties required under Section 22603 of the Business and Professions Code.
(2)Any guidance issued pursuant to paragraph (1) shall, at a minimum, be consistent with guidance issued by the U.S. Artificial Intelligence Safety Institute and the National Institute of Standards and Technology.
(g)Regulations and guidance adopted pursuant to this section shall be approved by the Board of Frontier Models before taking effect.
SEC. 5.Section 11547.6.1 is added to the Government Code, to read:11547.6.1.(a)There is hereby established in the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2026, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations on potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations on the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following:
(A)Representatives of the University of California and other public and private academic research institutions and national laboratories.
(B)Representatives of impacted workforce labor organizations.
(C)Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
(D)Experts in technology and artificial intelligence to provide technical assistance.
(E)Personnel from other relevant departments and agencies as necessary.
(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President Pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.
(h)If CalCompute is established within the University of California pursuant to subdivision (c), the University of California may receive private donations for the purposes of implementing CalCompute.
(i)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.The Legislature finds and declares that Section 3 of this act, which adds Chapter 22.6 (commencing with Section 22602) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in unredacted safety and security protocols and auditors reports may contain corporate proprietary information or information about covered models and covered model derivatives that could threaten public safety if disclosed to the public.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Artificial intelligence safety incident means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
(1)A covered model or covered model derivative autonomously engaging in behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model or covered model derivative.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model or covered model derivative.
(4)Unauthorized use of a covered model or covered model derivative to cause or materially enable critical harm.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)(1)Covered model means either of the following:
(A)Before January 1, 2027, covered model means either of the following:
(i)An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market prices of cloud compute at the start of training as reasonably assessed by the developer.
(ii)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than three times 10^25 integer or floating-point operations, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(B)(i)Except as provided in clause (ii), on and after January 1, 2027, covered model means any of the following:
(I)An artificial intelligence model trained using a quantity of computing power determined by the Government Operations Agency pursuant to Section 11547.6 of the Government Code, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
(II)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Government Operations Agency, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(ii)If the Government Operations Agency does not adopt a regulation governing subclauses (I) and (II) of clause (i) before January 1, 2027, the definition of covered model in subparagraph (A) shall be operative until the regulation is adopted.
(2)On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(f)Covered model derivative means any of the following:
(1)An unmodified copy of a covered model.
(2)A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
(3)(A)(i)Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding three times 10^25 integer or floating point operations, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(ii)On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Government Operations Agency, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(B)If the Government Operations Agency does not adopt a regulation governing clause (ii) of subparagraph (A) by January 1, 2027, the quantity of computing power specified in clause (i) of subparagraph (A) shall continue to apply until the regulation is adopted.
(4)A copy of a covered model that has been combined with other software.
(g)(1)Critical harm means any of the following harms caused or materially enabled by a covered model or covered model derivative:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model conducting, or providing precise instructions for conducting, a cyberattack or series of cyberattacks on critical infrastructure.
(C)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that does both of the following:
(i)Acts with limited human oversight, intervention, or supervision.
(ii)Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
(D)Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2)Critical harm does not include any of the following:
(A)Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.
(B)Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other softwares ability to cause or materially enable the harm.
(C)Harms that are not caused or materially enabled by the developers creation, storage, use, or release of a covered model or covered model derivative.
(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)Developer means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power and cost, or by fine-tuning an existing covered model or covered model derivative using a quantity of computing power and cost greater than the amount specified in subdivision (e).
(j)Fine-tuning means adjusting the model weights of a trained covered model or covered model derivative by exposing it to additional data.
(k)Full shutdown means the cessation of operation of all of the following:
(1)The training of a covered model.
(2)A covered model controlled by a developer.
(3)All covered model derivatives controlled by a developer.
(l)Model weight means a numerical parameter in an artificial intelligence model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(m)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(n)Post-training modification means modifying the capabilities of a covered model or covered model derivative by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(o)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models and covered model derivatives across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of covered model derivatives.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model and covered model derivatives.
22603.(a)Before beginning to initially train a covered model, the developer shall do all of the following:
(1)Implement reasonable administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)(A)Implement the capability to promptly enact a full shutdown.
(B)When enacting a full shutdown, the developer shall take into account, as appropriate, the risk that a shutdown of the covered model, or particular covered model derivatives, could cause disruptions to critical infrastructure.
(3)Implement a written and separate safety and security protocol that does all of the following:
(A)Specifies protections and procedures that, if successfully implemented, would successfully comply with the developers duty to take reasonable care to avoid producing a covered model or covered model derivative that poses an unreasonable risk of causing or materially enabling a critical harm.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies a testing procedure, which takes safeguards into account as appropriate, that takes reasonable care to evaluate if both of the following are true:
(i)A covered model poses an unreasonable risk of causing or enabling a critical harm.
(ii)Covered model derivatives pose an unreasonable risk of causing or enabling a critical harm.
(D)Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
(E)Describes in detail how the testing procedure addresses the possibility that a covered model or covered model derivative can be used to make post-training modifications or create another covered model in a manner that may cause or materially enable a critical harm.
(F)Describes in detail how the developer will fulfill their obligations under this chapter.
(G)Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.
(H)Describes in detail the conditions under which a developer would enact a full shutdown.
(I)Describes in detail the procedure by which the safety and security protocol may be modified.
(4)Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, or any covered model derivatives controlled by the developer, monitoring and reporting on implementation.
(5)Retain an unredacted copy of the safety and security protocol for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years, including records and dates of any updates or revisions.
(6)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(7)(A)(i)Conspicuously publish a copy of the redacted safety and security protocol and transmit a copy of the redacted safety and security protocol to the Attorney General.
(ii)A redaction in the safety and security protocol may be made only if the redaction is reasonably necessary to protect any of the following:
(I)Public safety.
(II)Trade secrets, as defined in Section 3426.1 of the Civil Code.
(III)Confidential information pursuant to state and federal law.
(B)The developer shall grant to the Attorney General access to the unredacted safety and security protocol upon request.
(C)A safety and security protocol disclosed to the Attorney General pursuant to this paragraph is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(D)If the safety and security protocol is materially modified, conspicuously publish and transmit to the Attorney General an updated redacted copy within 30 days of the modification.
(8)Take reasonable care to implement other appropriate measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or materially enabling critical harms.
(b)Before using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or before making a covered model or covered model derivative available for commercial or public, or foreseeably public, use, the developer of a covered model shall do all of the following:
(1)Assess whether the covered model is reasonably capable of causing or materially enabling a critical harm.
(2)Record, as and when reasonably possible, and retain for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years information on the specific tests and test results used in the assessment pursuant to paragraph (1) that provides sufficient detail for third parties to replicate the testing procedure.
(3)Take reasonable care to implement appropriate safeguards to prevent the covered model and covered model derivatives from causing or materially enabling a critical harm.
(4)Take reasonable care to ensure, to the extent reasonably possible, that the covered models actions and the actions of covered model derivatives, as well as critical harms resulting from their actions, can be accurately and reliably attributed to them.
(c)A developer shall not use a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or make a covered model or a covered model derivative available for commercial or public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative will cause or materially enable a critical harm.
(d)A developer of a covered model shall annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section.
(e)(1)Beginning January 1, 2026, a developer of a covered model shall annually retain a third-party auditor that conducts audits consistent with best practices for auditors to perform an independent audit of compliance with the requirements of this section.
(2)An auditor shall conduct audits consistent with regulations issued by the Government Operations Agency pursuant to subdivision (d) of Section 11547.6 of the Government Code.
(3)The auditor shall be granted access to unredacted materials as necessary to comply with the auditors obligations under this subdivision.
(4)The auditor shall produce an audit report including all of the following:
(A)A detailed assessment of the developers steps to comply with the requirements of this section.
(B)If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
(C)A detailed assessment of the developers internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
(D)The signature of the lead auditor certifying the results of the auditor.
(5)The developer shall retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years.
(6)(A)(i)The developer shall conspicuously publish a redacted copy of the auditors report and transmit to the Attorney General a copy of the redacted auditors report.
(ii)A redaction in the auditors report may be made only if the redaction is reasonably necessary to protect any of the following:
(I)Public safety.
(II)Trade secrets, as defined in Section 3426.1 of the Civil Code.
(III)Confidential information pursuant to state and federal law.
(B)The developer shall grant to the Attorney General access to the unredacted auditors report upon request.
(C)An auditors report disclosed to the Attorney General pursuant to this paragraph is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(7)An auditor shall not knowingly make a material misrepresentation in the auditors report.
(f)(1)(A)A developer of a covered model shall annually submit to the Attorney General a statement of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, that meets the requirements of paragraph (2).
(B)This paragraph applies if the covered model or any covered model derivatives controlled by the developer remain in commercial or public use or remain available for commercial or public use.
(2)In a statement submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)An assessment of the nature and magnitude of critical harms that the covered model or covered model derivatives may reasonably cause or materially enable and the outcome of the assessment required by paragraph (1) of subdivision (b).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent the covered model or covered model derivatives from causing or materially enabling critical harms.
(C)A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the statement and the contact information of any third parties relied upon to validate compliance.
(g)A developer of a covered model shall report each artificial intelligence safety incident affecting the covered model, or any covered model derivatives controlled by the developer, to the Attorney General within 72 hours of the developer learning of the artificial intelligence safety incident or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)A developer shall submit to the Attorney General a statement described by subdivision (f) no more than 30 days after using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or covered model derivative available for commercial or public, or foreseeably public, use for the first time.
(2)This subdivision does not apply with respect to a covered model derivative if the developer submitted a statement described by subdivision (f) for the applicable covered model from which the covered model derivative is derived.
(i)In fulfilling its obligations under this chapter, a developer shall consider industry best practices and applicable guidance from the U.S. Artificial Intelligence Safety Institute, National Institute of Standards and Technology, the Government Operations Agency, and other reputable standard-setting organizations.
(j)(1)This section shall not apply to products or services to the extent that the requirements would strictly conflict with the terms of a contract with a federal government entity and a developer of a covered model.
(2)This section applies to the development, use, or commercial or public release of a covered model or covered model derivative for any use that is not the subject of a contract with a federal government entity, even if that covered model or covered model derivative has already been developed, trained, or used by a federal government entity.
22604.(a)A person that operates a computing cluster shall implement written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(1)Obtain the prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(A)The identity of the prospective customer.
(B)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(C)The email address and telephonic contact information used to verify the prospective customers identity.
(2)Assess whether the prospective customer intends to utilize the computing cluster to train a covered model.
(3)If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to paragraph (1) and conduct the assessment required pursuant to paragraph (2) prior to each utilization.
(4)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(5)Maintain for seven years and provide to the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(6)Implement the capability to promptly enact a full shutdown of any resources being used to train or operate models under the customers control.
(b)A person that operates a computing cluster shall consider industry best practices and applicable guidance from the U.S. Artificial Intelligence Safety Institute, National Institute of Standards and Technology, and other reputable standard-setting organizations.
(c)In complying with the requirements of this section, a person that operates a computing cluster may impose reasonable requirements on customers to prevent the collection or retention of personal information that the person that operates a computing cluster would not otherwise collect or retain, including a requirement that a corporate customer submit corporate contact information rather than information that would identify a specific individual.
22606.(a)The Attorney General may bring a civil action for a violation of this chapter and to recover all of the following:
(1)For a violation that causes death or bodily harm to another human, harm to property, theft or misappropriation of property, or that constitutes an imminent risk or threat to public safety that occurs on or after January 1, 2026, a civil penalty in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(2)For a violation of Section 22607 that would constitute a violation of the Labor Code, a civil penalty specified in subdivision (f) of Section 1102.5 of the Labor Code.
(3)For a person that operates a computing cluster for a violation of Section 22604, for an auditor for a violation of paragraph (6) of subdivision (e) of Section 22603, or for an auditor who intentionally or with reckless disregard violates a provision of subdivision (e) of Section 22603 other than paragraph (6) or regulations issued by the Government Operations Agency pursuant to Section 11547.6 of the Government Code, a civil penalty in an amount not exceeding fifty thousand dollars ($50,000) for a first violation of Section 22604, not exceeding one hundred thousand dollars ($100,000) for any subsequent violation, and not exceeding ten million dollars ($10,000,000) in the aggregate for related violations.
(4)Injunctive or declaratory relief.
(5)(A)Monetary damages.
(B)Punitive damages pursuant to subdivision (a) of Section 3294 of the Civil Code.
(6)Attorneys fees and costs.
(7)Any other relief that the court deems appropriate.
(b)In determining whether the developer exercised reasonable care as required in Section 22603, all of the following considerations are relevant but not conclusive:
(1)The quality of a developers safety and security protocol.
(2)The extent to which the developer faithfully implemented and followed its safety and security protocol.
(3)Whether, in quality and implementation, the developers safety and security protocol was inferior, comparable, or superior to those of developers of comparably powerful models.
(4)The quality and rigor of the developers investigation, documentation, evaluation, and management of risks of critical harm posed by its model.
(c)(1)A provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of this chapter, or to shift that liability to any person or entity in exchange for their use or access of, or right to use or access, a developers products or services, including by means of a contract of adhesion, is void as a matter of public policy.
(2)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section to the maximum extent allowed by law if the court concludes that both of the following are true:
(A)The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
(B)As the result of the steps described in subparagraph (A), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties, damages, or injunctive relief under this section.
(d)Penalties collected pursuant to this section by the Attorney General shall be deposited into the Public Rights Law Enforcement Special Fund established pursuant to Section 12530 of the Government Code.
(e)This section does not limit the application of other laws.
22607.(a)A developer of a covered model or a contractor or subcontractor of the developer shall not do any of the following:
(1)Prevent an employee from disclosing information to the Attorney General or the Labor Commissioner, including through terms and conditions of employment or seeking to enforce terms and conditions of employment if the employee has reasonable cause to believe the information indicates either of the following:
(A)The developer is out of compliance with the requirements of Section 22603.
(B)An artificial intelligence model, including a model that is not a covered model or a covered model derivative, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
(2)Retaliate against an employee for disclosing information to the Attorney General or the Labor Commissioner pursuant to paragraph (1).
(3)Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 (commencing with Section 16600) of Division 7 or any other provision of state law.
(b)An employee harmed by a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
(c)(1)The Attorney General or Labor Commissioner may publicly release or provide to the Governor any complaint, or a summary of that complaint, pursuant to this section if the Attorney General or the Labor Commissioner concludes that doing so will serve the public interest.
(2)If the Attorney General or the Labor Commissioner publicly releases a complaint, or a summary of a complaint, pursuant to paragraph (1), the Attorney General or the Labor Commissioner shall redact from the complaint any information that is confidential or otherwise exempt from public disclosure pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code) and any information that the Attorney General or the Labor Commissioner determines would likely pose an unreasonable risk to public safety if it were disclosed to the public.
(d)A developer shall provide a clear notice to all employees working on covered models and covered model derivatives of their rights and responsibilities under this section, including the right of employees of contractors and subcontractors to use the developers internal process for making protected disclosures pursuant to subdivision (e). A developer is presumed to be in compliance with the requirements of this subdivision if the developer does either of the following:
(1)At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice, and ensure that employees who work remotely periodically receive an equivalent notice.
(2)No less frequently than once every year, provides written notice to all employees of their rights and responsibilities under this chapter and ensures that the notice is received and acknowledged by all of those employees.
(e)(1)(A)A developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer has violated any provision of Section 22603 or any other law, or has made false or materially misleading statements related to its safety and security protocol, or failed to disclose known risks to employees, including, at a minimum, a monthly update to the person who made the disclosure regarding the status of the developers investigation of the disclosure and the actions taken by the developer in response to the disclosure.
(B)The process required by this paragraph shall apply to employees of the developers contractors and subcontractors working on covered models and covered model derivatives and allow those employees to disclose the same information to the developer that an employee of the developer may disclose and provide the same anonymity and protections against retaliation to the employees of the contractor or subcontractor that apply to disclosures by employees of the developer.
(2)The disclosures and responses of the process required by this subdivision shall be maintained for a minimum of seven years from the date when the disclosure or response is created. Each disclosure and response shall be shared with officers and directors of the developer whose acts or omissions are not implicated by the disclosure or response no less frequently than once per quarter. In the case of a report or disclosure regarding alleged misconduct by a contractor or subcontractor, the developer shall notify the officers and directors of the contractor or subcontractor whose acts or omissions are not implicated by the disclosure or response about the status of their investigation no less frequently than once per quarter.
(f)This section does not limit protections provided to employees by Section 1102.5 of the Labor Code, Section 12964.5 of the Government Code, or other law.
(g)As used in this section:
(1)Employee has the same meaning as defined in Section 1132.4 of the Labor Code and includes both of the following:
(A)Contractors or subcontractors and unpaid advisors involved with assessing, managing, or addressing the risk of critical harm from covered models and covered model derivatives.
(B)Corporate officers.
(2)Contractor or subcontractor has the same meaning as in Section 1777.1 of the Labor Code.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
22609.This chapter does not apply to the extent that it is preempted by federal law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section, critical harm has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)There is hereby established the Board of Frontier Models. The board shall be housed in the Government Operations Agency and shall be independent of the Department of Technology. The Governor may appoint an executive officer of the board, subject to Senate confirmation, who shall hold the office at the pleasure of the Governor. The executive officer shall be the administrative head of the board and shall exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged.
(c)(1)Commencing January 1, 2026, the Board of Frontier Models shall be composed of nine members, as follows:
(1)(A)A member of the open-source community appointed by the Governor and subject to Senate confirmation.
(2)(B)A member of the artificial intelligence industry appointed by the Governor and subject to Senate confirmation.
(3)(C)An expert in chemical, biological, radiological, or nuclear weapons appointed by the Governor and subject to Senate confirmation.
(4)(D)An expert in artificial intelligence safety appointed by the Governor and subject to Senate confirmation.
(5)(E)An expert in cybersecurity of critical infrastructure appointed by the Governor and subject to Senate confirmation.
(6)(F)Two members who are academics with expertise in artificial intelligence appointed by the Speaker of the Assembly.
(7)(G)Two members appointed by the Senate Rules Committee.
(2)A member of the Board of Frontier Models shall meet all of the following criteria:
(A)A member shall be free of direct and indirect external influence and shall not seek or take instructions from another.
(B)A member shall not take an action or engage in an occupation, whether gainful or not, that is incompatible with the members duties.
(C)A member shall not, either at the time of the members appointment or during the members term, have a financial interest in an entity that is subject to regulation by the board.
(3)A member of the board shall serve at the pleasure of the members appointing authority but shall serve for no longer than eight consecutive years.
(d)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue regulations to update both of the following thresholds in the definition of a covered model to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose significant risk of causing or materially enabling critical harms.
(2)The updated definition shall contain both of the following:
(A)The initial compute threshold that an artificial intelligence model shall exceed to be considered a covered model.
(B)The fine-tuning compute threshold that an artificial intelligence model shall meet to be considered a covered model.
(3)In developing regulations pursuant to this subdivision, the Government Operations Agency shall take into account all of the following:
(A)The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or materially enable a critical harm.
(B)Similar thresholds used in federal law, guidance, or regulations for the management of artificial intelligence models with reasonable risks of causing or enabling critical harms.
(C)Input from stakeholders, including academics, industry, the open-source community, and government entities.
(e)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue regulations to establish binding auditing requirements applicable to audits conducted pursuant to subdivision (e) of Section 22603 of the Business and Professions Code to ensure the integrity, independence, efficiency, and effectiveness of the auditing process. In developing regulations pursuant to this subdivision, the Government Operations Agency shall take into account both of the following:
(A)Relevant standards or requirements imposed under federal or state law or through self-regulatory or standards-setting bodies.
(B)Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
(2)Any regulations issued pursuant to paragraph (1) shall, at a minimum, be consistent with guidance issued by the U.S. Artificial Intelligence Safety Institute and the National Institute of Standards and Technology.
(f)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue guidance for preventing unreasonable risks of covered models and covered model derivatives causing or materially enabling critical harms, including, but not limited to, more specific components of, or requirements under, the duties required under Section 22603 of the Business and Professions Code.
(2)Any guidance issued pursuant to paragraph (1) shall, at a minimum, be consistent with guidance issued by the U.S. Artificial Intelligence Safety Institute and the National Institute of Standards and Technology.
(g)Regulations and guidance adopted pursuant to this section shall be approved by the Board of Frontier Models before taking effect.
SEC. 5.Section 11547.6.1 is added to the Government Code, to read:11547.6.1.(a)There is hereby established in the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2026, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations on potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations on the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following:
(A)Representatives of the University of California and other public and private academic research institutions and national laboratories.
(B)Representatives of impacted workforce labor organizations.
(C)Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
(D)Experts in technology and artificial intelligence to provide technical assistance.
(E)Personnel from other relevant departments and agencies as necessary.
(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President Pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.
(h)If CalCompute is established within the University of California pursuant to subdivision (c), the University of California may receive private donations for the purposes of implementing CalCompute.
(i)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.The Legislature finds and declares that Section 3 of this act, which adds Chapter 22.6 (commencing with Section 22602) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in unredacted safety and security protocols and auditors reports may contain corporate proprietary information or information about covered models and covered model derivatives that could threaten public safety if disclosed to the public.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Artificial intelligence safety incident means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
(1)A covered model or covered model derivative autonomously engaging in behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model. model or covered model derivative.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model. model or covered model derivative.
(4)Unauthorized use of a covered model or covered model derivative to cause or materially enable critical harm.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)(1)Covered model means either of the following:
(A)Before January 1, 2027, covered model means either of the following:
(i)An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market prices of cloud compute at the start of training as reasonably assessed by the developer.
(ii)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than three times 10^25 integer or floating-point operations. operations, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(B)(i)Except as provided in clause (ii), on and after January 1, 2027, covered model means any of the following:
(I)An artificial intelligence model trained using a quantity of computing power determined by the Frontier Model Division Government Operations Agency pursuant to Section 11547.6 of the Government Code, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
(II)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division. Government Operations Agency, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(ii)If the Frontier Model Division Government Operations Agency does not adopt a regulation governing subclauses (I) and (II) of clause (i) by before January 1, 2027, the definition of covered model in subparagraph (A) continues to shall be in effect operative until the regulation is adopted.
(2)On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(f)Covered model derivative means any of the following:
(1)An unmodified copy of a covered model.
(2)A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
(3)(A)(i)Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding three times 10^25 integer or floating point operations. operations, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(ii)On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Frontier Model Division. Government Operations Agency, the cost of which, as reasonably assessed by the developer, exceeds ten million dollars ($10,000,000) if calculated using the average market price of cloud compute at the start of fine-tuning.
(B)If the Frontier Model Division Government Operations Agency does not adopt a regulation governing clause (ii) of subparagraph (A) by January 1, 2027, the quantity of computing power specified in clause (i) of subparagraph (A) shall continue to apply until the regulation is adopted.
(4)A copy of a covered model that has been combined with other software.
(g)(1)Critical harm means any of the following harms caused or materially enabled by a covered model or covered model derivative:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model conducting, or providing precise instructions for conducting conducting, a cyberattack or series of cyberattacks on critical infrastructure.
(C)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that does both of the following:
(i)Acts with limited human oversight, intervention, or supervision.
(ii)Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
(D)Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2)Critical harm does not include either any of the following:
(A)Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model. model or covered model derivative.
(B)Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other softwares ability to cause or materially enable the harm.
(C)Harms that are not caused or materially enabled by the developers creation, storage, use, or release of a covered model or covered model derivative.
(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)Developer means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, power and cost, or by fine-tuning an existing covered model or covered model derivative using sufficient a quantity of computing power pursuant to and cost greater than the amount specified in subdivision (e).
(j)Fine-tuning means adjusting the model weights of a trained covered model or covered model derivative by exposing it to additional data.
(k)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.(l)(k)Full shutdown means the cessation of operation of any all of the following:
(1)The training of a covered model.
(2)A covered model. model controlled by a developer.
(3)All covered model derivatives controlled by a developer.
(m)(l)Model weight means a numerical parameter in an artificial intelligence model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(n)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and that may be freely modified and redistributed.(o)(m)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(p)(n)Post-training modification means modifying the capabilities of a covered model or covered model derivative by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(q)Reasonable assurance does not mean full certainty or practical certainty.(r)(o)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models and covered model derivatives across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of covered model derivatives.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model. model and covered model derivatives.
22603.(a)Before a developer initially trains beginning to initially train a covered model, the developer shall do all of the following:
(1)Implement reasonable administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)(A)Implement the capability to promptly enact a full shutdown.
(B)When enacting a full shutdown, the developer shall take into account, as appropriate, the risk that a shutdown of the covered model, or particular covered model derivatives, could cause disruptions to critical infrastructure.
(3)Implement a written and separate safety and security protocol that does all of the following:
(A)If a developer complies with the safety and security protocol, provides reasonable assurance that the developer will not produce Specifies protections and procedures that, if successfully implemented, would successfully comply with the developers duty to take reasonable care to avoid producing a covered model or covered model derivative that poses an unreasonable risk of causing or materially enabling a critical harm.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to provide reasonable assurance of both of the following:(C)Identifies a testing procedure, which takes safeguards into account as appropriate, that takes reasonable care to evaluate if both of the following are true:
(i)That a A covered model does not pose poses an unreasonable risk of causing or enabling a critical harm.
(ii)That covered Covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
(D)Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
(E)Describes in detail how the testing procedure addresses the possibility that a covered model or covered model derivative can be used to make post-training modifications or create another covered model in a manner that may generate hazardous capabilities. cause or materially enable a critical harm.
(F)Provides sufficient detail for third parties to replicate the testing procedure.(G)(F)Describes in detail how the developer will fulfill their obligations under this chapter.
(H)(G)Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.
(I)(H)Describes in detail the conditions under which a developer would enact a full shutdown.
(J)(I)Describes in detail the procedure by which the safety and security protocol may be modified.
(4)Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, or any covered model derivatives controlled by the developer, monitoring and reporting on implementation.
(5)Provide a Retain an unredacted copy of the safety and security protocol to the Frontier Model Division. for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years, including records and dates of any updates or revisions.
(6)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(7)(A)(i)Conspicuously publish a copy of the redacted safety and security protocol and transmit a copy of the redacted safety and security protocol to the Attorney General.
(ii)A redaction in the safety and security protocol may be made only if the redaction is reasonably necessary to protect any of the following:
(I)Public safety.
(II)Trade secrets, as defined in Section 3426.1 of the Civil Code.
(III)Confidential information pursuant to state and federal law.
(B)The developer shall grant to the Attorney General access to the unredacted safety and security protocol upon request.
(C)A safety and security protocol disclosed to the Attorney General pursuant to this paragraph is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(7)(D)If the safety and security protocol is materially modified, provide conspicuously publish and transmit to the Attorney General an updated redacted copy to the Frontier Model Division within 10 business days. within 30 days of the modification.
(8)Implement Take reasonable care to implement other reasonable appropriate measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or materially enabling critical harms.
(b)Before using a covered model or covered model derivative, derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or before making a covered model or covered model derivative available for commercial or public public, or foreseeably public, use, the developer of a covered model shall do all of the following:
(1)Assess whether the covered model is reasonably capable of causing or materially enabling a critical harm.
(2)Record, as and when reasonably possible, and retain for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years information on the specific tests and test results used in the assessment pursuant to paragraph (1) that provides sufficient detail for third parties to replicate the testing procedure.
(2)Implement(3)Take reasonable care to implement appropriate safeguards to prevent the covered model and covered model derivatives from causing or materially enabling a critical harm.
(3)Ensure,(4)Take reasonable care to ensure, to the extent reasonably possible, that the covered models actions and the actions of covered model derivatives, as well as critical harms resulting from their actions, can be accurately and reliably attributed to them.
(c)A developer shall not use a covered model commercially or publicly, or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or make a covered model or a covered model derivative available for commercial or public public, or foreseeably public, use, if there is an unreasonable risk that the covered model or covered model derivative can will cause or materially enable a critical harm.
(d)A developer of a covered model shall annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section.
(e)(1)Beginning January 1, 2028, 2026, a developer of a covered model shall annually retain a third-party auditor that conducts audits consistent with best practices for auditors to perform an independent audit of compliance with the requirements of this section.
(2)An auditor shall conduct audits consistent with regulations issued by the Government Operations Agency pursuant to subdivision (d) of Section 11547.6 of the Government Code.
(3)The auditor shall be granted access to unredacted materials as necessary to comply with the auditors obligations under this subdivision.
(2)(4)The auditor shall produce an audit report including all of the following:
(A)A detailed assessment of the developers steps to comply with the requirements of this section.
(B)If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
(C)A detailed assessment of the developers internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
(D)The signature of the lead auditor certifying the results of the auditor.
(5)The developer shall retain an unredacted copy of the audit report for as long as the covered model is made available for commercial, public, or foreseeably public use plus five years.
(6)(A)(i)The developer shall conspicuously publish a redacted copy of the auditors report and transmit to the Attorney General a copy of the redacted auditors report.
(ii)A redaction in the auditors report may be made only if the redaction is reasonably necessary to protect any of the following:
(I)Public safety.
(II)Trade secrets, as defined in Section 3426.1 of the Civil Code.
(III)Confidential information pursuant to state and federal law.
(B)The developer shall grant to the Attorney General access to the unredacted auditors report upon request.
(C)An auditors report disclosed to the Attorney General pursuant to this paragraph is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(7)An auditor shall not knowingly make a material misrepresentation in the auditors report.
(f)(1)(A)A developer of a covered model shall annually submit to the Frontier Model Division Attorney General a certification under penalty of perjury statement of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. This paragraph applies as long as the covered model or any covered model derivatives controlled by the developer remain in commercial or public use, or remain available for commercial or public use. that meets the requirements of paragraph (2).
(B)This paragraph applies if the covered model or any covered model derivatives controlled by the developer remain in commercial or public use or remain available for commercial or public use.
(2)In a certification statement submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The An assessment of the nature and magnitude of critical harms that the covered model or covered model derivatives may reasonably cause or enable, materially enable and the outcome of the assessment required by paragraph (1) of subdivision (b).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent the covered model or covered model derivatives from causing or materially enabling critical harms.
(C)A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the certification, statement and the contact information of any third parties relied upon to validate compliance.
(D)Beginning January 1, 2028, the most recent audit report pursuant to subdivision (e).(g)A developer of a covered model shall report each artificial intelligence safety incident affecting the covered model, or any covered model derivatives controlled by the developer, to the Frontier Model Division Attorney General within 72 hours of the developer learning of the artificial intelligence safety incident, incident or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)A developer shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section Attorney General a statement described by subdivision (f) no more than 30 days after using a covered model or covered model derivative for a purpose not exclusively related to the training or reasonable evaluation of the covered model or compliance with state or federal law or making a covered model or covered model derivative available for commercial or public public, or foreseeably public, use for the first time. A developer need not submit a certification for a covered model derivative if the developer has already submitted a certification for the applicable covered model.
(2)This subdivision does not apply with respect to a covered model derivative if the developer submitted a statement described by subdivision (f) for the applicable covered model from which the covered model derivative is derived.
(i)In fulfilling their its obligations under this chapter, a developer shall consider industry best practices and applicable guidance from the Frontier Model Division, U.S. Artificial Intelligence Safety Institute, National Institute of Standards and Technology, the Government Operations Agency, and other reputable standard-setting organizations.
(j)(1)This section shall not apply to products or services to the extent that the requirements would strictly conflict with the terms of a contract with a federal government entity and a developer of a covered model.
(2)This section applies to the development, use, or commercial or public release of a covered model or covered model derivative for any use that is not the subject of a contract with a federal government entity, even if that covered model or covered model derivative has already been developed, trained, or used by a federal government entity.
22604.(a)A person that operates a computing cluster shall implement written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(1)Obtain a the prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(A)The identity of that the prospective customer.
(B)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(C)The email address and telephonic contact information used to verify a the prospective customers identity.
(2)Assess whether a the prospective customer intends to utilize the computing cluster to train a covered model.
(3)If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to paragraph (1) and conduct the assessment required pursuant to paragraph (2) prior to each utilization.
(4)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(5)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(6)Implement the capability to promptly enact a full shutdown of any resources being used to train or operate models under the customers control.
(b)A person that operates a computing cluster shall consider industry best practices and applicable guidance from the Frontier Model Division, U.S. Artificial Intelligence Safety Institute, National Institute of Standards and Technology, and other reputable standard-setting organizations.
(c)In complying with the requirements of this section, a person that operates a computing cluster may impose reasonable requirements on customers to prevent the collection or retention of personal information that the person that operates a computing cluster would not otherwise collect or retain, including a requirement that a corporate customer submit corporate contact information rather than information that would identify a specific individual.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.22606.(a)The following are unlawful acts:(1)For a developer to fail to comply with any of the requirements of Section 22603 or subdivision (a) of Section 22605.(2)For a person who operates a computing cluster to fail to comply with the requirements of Section 22604 or subdivision (b) of Section 22605.(3)For a developer to fail to comply with any of the requirements of Section 22607.(b)The following parties may bring a civil action pursuant to subdivision (a):(1)The Attorney General to enforce any provision of this chapter.(2)The Labor Commissioner to enforce any provision of Section 22607 that would constitute a violation of the Labor Code.(c)22606.(a)The parties listed in subdivision (b) are entitled Attorney General may bring a civil action for a violation of this chapter and to recover all of the following in addition to any civil penalties specified in this chapter: following:
(1)A civil penalty for a violation For a violation that causes death or bodily harm to another human, harm to property, theft or misappropriation of property, or that constitutes an imminent risk or threat to public safety that occurs on or after January 1, 2026, a civil penalty in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(2)For a violation of Section 22607 that would constitute a violation of the Labor Code, a civil penalty specified in subdivision (f) of Section 1102.5 of the Labor Code.
(3)For a person that operates a computing cluster for a violation of Section 22604, for an auditor for a violation of paragraph (6) of subdivision (e) of Section 22603, or for an auditor who intentionally or with reckless disregard violates a provision of subdivision (e) of Section 22603 other than paragraph (6) or regulations issued by the Government Operations Agency pursuant to Section 11547.6 of the Government Code, a civil penalty in an amount not exceeding fifty thousand dollars ($50,000) for a first violation of Section 22604, not exceeding one hundred thousand dollars ($100,000) for any subsequent violation, and not exceeding ten million dollars ($10,000,000) in the aggregate for related violations.
(2)(A)(4)Injunctive or declaratory relief, including, but not limited to, orders to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer. relief.
(B)The court may only order relief under this paragraph for a covered model that has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.(3)(5)(A)Monetary damages.
(B)Punitive damages pursuant to subdivision (a) of Section 3294 of the Civil Code.
(4)(6)Attorneys fees and costs.
(5)(7)Any other relief that the court deems appropriate.
(b)In determining whether the developer exercised reasonable care as required in Section 22603, all of the following considerations are relevant but not conclusive:
(1)The quality of a developers safety and security protocol.
(2)The extent to which the developer faithfully implemented and followed its safety and security protocol.
(3)Whether, in quality and implementation, the developers safety and security protocol was inferior, comparable, or superior to those of developers of comparably powerful models.
(4)The quality and rigor of the developers investigation, documentation, evaluation, and management of risks of critical harm posed by its model.
(d)(c)(1)A provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of this chapter, or to shift that liability to any person or entity in exchange for their use or access of, or right to use or access, a developers products or services, including by means of a contract of adhesion, is void as a matter of public policy.
(2)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section to the maximum extent allowed by law if the court concludes that both of the following are true:
(A)The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
(B)As the result of the steps described in subparagraph (A), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties penalties, damages, or injunctive relief under this section.
(d)Penalties collected pursuant to this section by the Attorney General shall be deposited into the Public Rights Law Enforcement Special Fund established pursuant to Section 12530 of the Government Code.
(e)This section does not limit the application of other laws.
22607.(a)A developer of a covered model or a contractor or subcontractor of the developer shall not do either any of the following:
(1)Prevent an employee from disclosing information to the Attorney General or the Labor Commissioner, including through terms and conditions of employment or seeking to enforce terms and conditions of employment, employment if the employee has reasonable cause to believe the information indicates either of the following:
(A)The developer is out of compliance with the requirements of Section 22603.
(B)An artificial intelligence model, including a model that is not a covered model, model or a covered model derivative, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
(2)Retaliate against an employee for disclosing information to the Attorney General or the Labor Commissioner, if the employee has reasonable cause to believe either subparagraph (A) or (B) of Commissioner pursuant to paragraph (1).
(3)Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 (commencing with Section 16600) of Division 7 or any other provision of state law.
(b)(1)An employee harmed by a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
(2)An employee of the Frontier Model Division may report any violation of this chapter by the Frontier Model Division to the State Auditor pursuant to the provisions of the California Whistleblower Protection Act (Article 3 (commencing with Section 8547) of Chapter 6.5 of Division 1 of Title 2 of the Government Code) which shall govern any such report.(c)(1)The Attorney General or Labor Commissioner may publicly release or provide, to the Frontier Model Division or the Governor, provide to the Governor any complaint, or a summary of that complaint, pursuant to this section if they conclude the Attorney General or the Labor Commissioner concludes that doing so will serve the public interest.
(2)If the Attorney General or the Labor Commissioner publicly releases a complaint, or a summary of a complaint, pursuant to paragraph (1), the Attorney General or the Labor Commissioner shall redact from the complaint any information that is confidential or otherwise exempt from public disclosure pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code) and any information that the Attorney General or the Labor Commissioner determines would likely pose an unreasonable risk to public safety if it were disclosed to the public.
(d)A developer and any contractor or subcontractor of the developer shall provide a clear notice to all employees working on covered models and covered model derivatives of their rights and responsibilities under this section. section, including the right of employees of contractors and subcontractors to use the developers internal process for making protected disclosures pursuant to subdivision (e). A developer is presumed to be in compliance with the requirements of this subdivision if the developer does one either of the following:
(1)At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice, and ensure that employees who work remotely periodically receive an equivalent notice.
(2)No less frequently than once every six months, provide year, provides written notice to all employees of their rights and responsibilities under this chapter and ensure ensures that such the notice is received and acknowledged by all of those employees.
(e)(1)(A)A developer and any contractor or subcontractor of the developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer has violated any provision of Section 22603 or any other law, or has made false or materially misleading statements related to its safety and security protocol, or failed to disclose known risks to employees, including, at a minimum, a monthly update to the disclosing employee person who made the disclosure regarding the status of the employees developers investigation of the disclosure and the actions taken by the developer, contractor, or subcontractor developer in response to the disclosure.
(B)The process required by this paragraph shall apply to employees of the developers contractors and subcontractors working on covered models and covered model derivatives and allow those employees to disclose the same information to the developer that an employee of the developer may disclose and provide the same anonymity and protections against retaliation to the employees of the contractor or subcontractor that apply to disclosures by employees of the developer.
(2)The disclosures and responses of the process required by this subdivision shall be maintained for a minimum of seven years from the date when the disclosure or response is created. Each disclosure and response shall be shared with officers and directors of the developer and any contractor or subcontractor of the developer whose acts or omissions are not implicated by the disclosure or response no less frequently than once per quarter. In the case of a report or disclosure regarding alleged misconduct by a contractor or subcontractor, the developer shall notify the officers and directors of the contractor or subcontractor whose acts or omissions are not implicated by the disclosure or response about the status of their investigation no less frequently than once per quarter.
(f)Nothing in this section shall be construed to This section does not limit protections provided to employees by Section 1102.5 of the Labor Code, Section 12964.5 of the Government Code, or other provisions of California law. law.
(g)As used in this section, the following definitions apply: section:
(1)Employee has the same meaning as defined in Section 1132.4 of the Labor Code and includes both of the following:
(A)Contractors or subcontractors, subcontractors and unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models. the risk of critical harm from covered models and covered model derivatives.
(B)Corporate officers.
(2)Contractor or subcontractor has the same meaning as in Section 1777.1 of the Labor Code.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
22609.This chapter does not apply to the extent that it is preempted by federal law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section, critical harm has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)There is hereby established the Board of Frontier Models. The board shall be housed in the Government Operations Agency and shall be independent of the Department of Technology. The Governor may appoint an executive officer of the board, subject to Senate confirmation, who shall hold the office at the pleasure of the Governor. The executive officer shall be the administrative head of the board and shall exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged.
(c)Commencing January 1, 2026, the Board of Frontier Models shall be composed of five nine members, as follows:
(1)A member of the open-source community, community appointed by the Governor, Governor and subject to Senate confirmation.
(2)A member of the artificial intelligence industry, industry appointed by the Governor, Governor and subject to Senate confirmation.
(3)A member of academia, appointed by the Governor, subject to Senate confirmation.(3)An expert in chemical, biological, radiological, or nuclear weapons appointed by the Governor and subject to Senate confirmation.
(4)An expert in artificial intelligence safety appointed by the Governor and subject to Senate confirmation.
(5)An expert in cybersecurity of critical infrastructure appointed by the Governor and subject to Senate confirmation.
(4)A member(6)Two members appointed by the Speaker of the Assembly.
(5)A member(7)Two members appointed by the Senate Rules Committee.
(d)The Frontier Model Division is hereby created within the Government Operations Agency under the direct supervision of the Board of Frontier Models.(e)The Frontier Model Division shall do all of the following:(1)Annually review certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.(3)(A)Issue guidance, standards, and best practices necessary to prevent unreasonable risks of covered models and covered model derivatives causing or enabling critical harms, including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.(B)Issue guidance regarding best practices for conducting an audit pursuant to subdivision (e) of Section 22603 of the Business and Professions Code.(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.(5)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.(6)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.(7)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that lack the ability to cause or enable critical harms.(B)Advise the Legislature on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.(8)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.(9)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt consistent with its policies and procedures for the promulgation of jury instructions.(B)In developing the proposed model jury instructions required by subparagraph (A), the Frontier Model Division shall consider and incorporate all of the following factors into the proposal that it submits to the Judicial Council:(i)All of the actions that a developer of a covered model must take pursuant to Section 22603 of the Business and Professions Code.(ii)How any regulations of the Frontier Model Division should be incorporated into the proposed model jury instructions.(iii)The rigor and quality of the safety and security protocol that a developer is required to implement while training and releasing its artificial intelligence model, and how to determine whether this safety and security protocol was inferior, comparable, or superior to the safety and security protocols of similarly situated developers.(iv)The rigor and quality of the developers investigation, documentation, evaluation, and management of its models potential hazardous capabilities, and associated risks.(10)(A)On or before January 1, 2027, and annually thereafter, issue regulations to update the definition of a covered model to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of causing or enabling critical harms. The updated definition shall contain both of the following:(i)The initial compute threshold that an artificial intelligence model must exceed to be considered a covered model, as defined in Section 22602 of the Business and Professions Code.(ii)The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.(B)In developing regulations pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:(i)The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or enable a critical harm.(ii)Similar thresholds used in federal law, guidance, or regulations for the management of models with reasonable risks of causing or enabling critical harms.(iii)Input from stakeholders, including academics, industry, and government entities, including from the open-source community.(11)Every 24 months after initial publication of guidance under paragraphs (3), (5), and (10), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.(12)On and after January 1, 2026, annually publish the inflation-adjusted dollar amounts described in paragraph (3) of subdivision (g) and paragraph (2) of subdivision (e) of Section 22602 of the Business and Professions Code.(f)There is hereby created in the General Fund the Frontier Model Division Programs Fund.(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.(d)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue regulations to update both of the following thresholds in the definition of a covered model to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose significant risk of causing or materially enabling critical harms.
(2)The updated definition shall contain both of the following:
(A)The initial compute threshold that an artificial intelligence model shall exceed to be considered a covered model.
(B)The fine-tuning compute threshold that an artificial intelligence model shall meet to be considered a covered model.
(3)In developing regulations pursuant to this subdivision, the Government Operations Agency shall take into account all of the following:
(A)The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or materially enable a critical harm.
(B)Similar thresholds used in federal law, guidance, or regulations for the management of artificial intelligence models with reasonable risks of causing or enabling critical harms.
(C)Input from stakeholders, including academics, industry, the open-source community, and government entities.
(e)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue regulations to establish binding auditing requirements applicable to audits conducted pursuant to subdivision (e) of Section 22603 of the Business and Professions Code to ensure the integrity, independence, efficiency, and effectiveness of the auditing process. In developing regulations pursuant to this subdivision, the Government Operations Agency shall take into account both of the following:
(A)Relevant standards or requirements imposed under federal or state law or through self-regulatory or standards-setting bodies.
(B)Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
(2)Any regulations issued pursuant to paragraph (1) shall, at a minimum, be consistent with guidance issued by the U.S. Artificial Intelligence Safety Institute and the National Institute of Standards and Technology.
(f)(1)On or before January 1, 2027, and annually thereafter, the Government Operations Agency shall issue guidance for preventing unreasonable risks of covered models and covered model derivatives causing or materially enabling critical harms, including, but not limited to, more specific components of, or requirements under, the duties required under Section 22603 of the Business and Professions Code.
(2)Any guidance issued pursuant to paragraph (1) shall, at a minimum, be consistent with guidance issued by the U.S. Artificial Intelligence Safety Institute and the National Institute of Standards and Technology.
(g)Regulations and guidance adopted pursuant to this section shall be approved by the Board of Frontier Models before taking effect.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:(1)A fully owned and hosted cloud platform.(2)Necessary human expertise to operate and maintain the platform.(3)Necessary human expertise to support, train, and facilitate use of CalCompute.(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.SEC. 5.Section 11547.6.1 is added to the Government Code, to read:11547.6.1.(a)There is hereby established in the Government Operations Agency a consortium that shall develop, pursuant to this section, a framework for the creation of a public cloud computing cluster to be known as CalCompute.
(b)The consortium shall develop a framework for creation of CalCompute that advances the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by doing, at a minimum, both of the following:
(1)Fostering research and innovation that benefits the public.
(2)Enabling equitable innovation by expanding access to computational resources.
(c)The consortium shall make reasonable efforts to ensure that CalCompute is established within the University of California to the extent possible.
(d)CalCompute shall include, but not be limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(e)The consortium shall operate in accordance with all relevant labor and workforce laws and standards.
(f)(1)On or before January 1, 2026, the Government Operations Agency shall submit, pursuant to Section 9795, a report from the consortium to the Legislature with the framework developed pursuant to subdivision (b) for creation and operation of CalCompute.
(2)The report required by this subdivision shall include all of the following elements:
(A)A landscape analysis of Californias current public, private, and nonprofit cloud computing platform infrastructure.
(B)An analysis of the cost to the state to build and maintain CalCompute and recommendations on potential funding sources.
(C)Recommendations for the governance structure and ongoing operation of CalCompute.
(D)Recommendations on the parameters for use of CalCompute, including, but not limited to, a process for determining which users and projects will be supported by CalCompute.
(E)An analysis of the states technology workforce and recommendations for equitable pathways to strengthen the workforce, including the role of CalCompute.
(F)A detailed description of any proposed partnerships, contracts, or licensing agreements with nongovernmental entities, including, but not limited to, technology-based companies, that demonstrates compliance with the requirements of subdivisions (c) and (d).
(G)Recommendations regarding how the creation and ongoing management of CalCompute can prioritize the use of the current public sector workforce.
(g)(1)The consortium shall, consistent with state constitutional law, consist of 14 members selected from among all of the following:
(A)Representatives of the University of California and other public and private academic research institutions and national laboratories.
(B)Representatives of impacted workforce labor organizations.
(C)Representatives of stakeholder groups with relevant expertise and experience, including, but not limited to, ethicists, consumer rights advocates, and other public interest advocates.
(D)Experts in technology and artificial intelligence to provide technical assistance.
(E)Personnel from other relevant departments and agencies as necessary.
(2)Eight members of the consortium shall be selected by the Secretary of Government Operations, and the President Pro Tempore of the Senate and the Speaker of the Assembly shall each select three members.
(h)If CalCompute is established within the University of California pursuant to subdivision (c), the University of California may receive private donations for the purposes of implementing CalCompute.
(i)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.SEC. 8.The Legislature finds and declares that Section 3 of this act, which adds Chapter 22.6 (commencing with Section 22602) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
Information in unredacted safety and security protocols and auditors reports may contain corporate proprietary information or information about covered models and covered model derivatives that could threaten public safety if disclosed to the public.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Artificial intelligence safety incident means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
(1)A covered model autonomously engaging in behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model.
(4)Unauthorized use of a covered model to cause or enable critical harm.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)(1)Covered model means either of the following:
(A)Before January 1, 2027, covered model means either of the following:
(i)An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market prices of cloud compute at the start of training as reasonably assessed by the developer.
(ii)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than three times 10^25 integer or floating-point operations.
(B)(i)Except as provided in clause (ii), on and after January 1, 2027, covered model means any of the following:
(I)An artificial intelligence model trained using a quantity of computing power determined by the Frontier Model Division pursuant to Section 11547.6 of the Government Code, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
(II)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division.
(ii)If the Frontier Model Division does not adopt a regulation governing subclauses (I) and (II) of clause (i) by January 1, 2027, the definition of covered model in subparagraph (A) continues to be in effect until the regulation is adopted.
(2)On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(f)Covered model derivative means any of the following:
(1)An unmodified copy of a covered model.
(2)A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
(3)(A)(i)Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding three times 10^25 integer or floating point operations.
(ii)On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Frontier Model Division.
(B)If the Frontier Model Division does not adopt a regulation governing clause (ii) of subparagraph (A) by January 1, 2027, the quantity of computing power specified in clause (i) of subparagraph (A) shall continue to apply until the regulation is adopted.
(4)A copy of a covered model that has been combined with other software.
(g)(1)Critical harm means any of the following harms caused or enabled by a covered model or covered model derivative:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure, occurring either in a single incident or over multiple related incidents. infrastructure by a model providing precise instructions for conducting a cyberattack or series of cyberattacks on critical infrastructure.
(C)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model autonomously engaging in conduct that would constitute a serious or violent felony under the Penal Code if undertaken by a human with the requisite mental state. does both of the following:
(i)Acts with limited human oversight, intervention, or supervision.
(ii)Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
(D)Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2)Critical harm does not include harms either of the following:
(A)Harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible. accessible from sources other than a covered model.
(B)Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other softwares ability to cause or materially enable the harm.
(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)Developer means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, or by fine-tuning an existing covered model using sufficient quantity of computing power pursuant to subdivision (e).
(j)Fine-tuning means adjusting the model weights of a trained covered model by exposing it to additional data.
(k)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(l)Full shutdown means the cessation of operation of any of the following:
(1)The training of a covered model.
(2)A covered model.
(3)All covered model derivatives controlled by a developer.
(m)Model weight means a numerical parameter in an artificial intelligence model that is adjusted through training and that helps determine how inputs are transformed into outputs.
(n)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and that may be freely modified and redistributed.
(o)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(p)Post-training modification means modifying the capabilities of a covered model by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(q)Reasonable assurance does not mean full certainty or practical certainty.
(r)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of covered model derivatives.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before a developer initially trains a covered model, the developer shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown.
(3)Implement a written and separate safety and security protocol that does all of the following:
(A)If a developer complies with the safety and security protocol, provides reasonable assurance that the developer will not produce a covered model or covered model derivative that poses an unreasonable risk of causing or enabling a critical harm.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to provide reasonable assurance of both of the following:
(i)That a covered model does not pose an unreasonable risk of causing or enabling a critical harm.
(ii)That covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
(D)Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
(E)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make post-training modifications or create another covered model in a manner that may generate hazardous capabilities.
(F)Provides sufficient detail for third parties to replicate the testing procedure.
(G)Describes in detail how the developer will fulfill their obligations under this chapter.
(H)Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.
(I)Describes in detail the conditions under which a developer would enact a full shutdown.
(J)Describes in detail the procedure by which the safety and security protocol may be modified.
(4)Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third-party auditors. implementation.
(5)Provide a copy of the safety and security protocol to the Frontier Model Division.
(6)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(7)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(8)Implement other reasonable measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or enabling critical harms.
(b)Before using a covered model or covered model derivative, or making a covered model or covered model derivative available for commercial or public use, the developer of a covered model shall do all of the following:
(1)Assess whether the covered model is reasonably capable of causing or enabling a critical harm.
(2)Implement reasonable safeguards to prevent the covered model and covered model derivatives from causing or enabling a critical harm.
(3)Ensure, to the extent reasonably possible, that the covered models actions and the actions of covered model derivatives, as well as critical harms resulting from their actions, can be accurately and reliably attributed to them.
(4)Beginning January 1, 2028, obtain a certificate of compliance from a third-party auditor who has been accredited pursuant to 11547.6 of the Government Code.(c)A developer shall not use a covered model commercially or publicly, or make a covered model or a covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.
(d)A developer of a covered model shall annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section.
(e)(1)Beginning January 1, 2028, a developer of a covered model shall annually retain a third-party auditor that conducts audits consistent with best practices for auditors to perform an independent audit of compliance with the requirements of this section.
(2)The auditor shall produce an audit report including all of the following:
(A)A detailed assessment of the developers steps to comply with the requirements of this section.
(B)If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
(C)A detailed assessment of the developers internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
(D)The signature of the lead auditor certifying the results of the auditor.
(e)(f)(1)A developer of a covered model shall annually submit to the Frontier Model Division a certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. This paragraph applies as long as the covered model or any covered model derivatives controlled by the developer remain in commercial or public use, or remain available for commercial or public use.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of critical harms that the covered model or covered model derivatives may reasonably cause or enable, and the outcome of the assessment required by paragraph (1) of subdivision (b).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent the covered model or covered model derivatives from causing or enabling critical harms.
(C)A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the certification, and the contact information of any third parties relied upon to validate compliance.
(D)Beginning January 1, 2028, a certificate of compliance from an accredited third-party auditor. the most recent audit report pursuant to subdivision (e).
(f)(g)A developer of a covered model shall report each artificial intelligence safety incident affecting the covered model, or any covered model derivatives controlled by the developer, to the Frontier Model Division within 72 hours of the developer learning of the artificial intelligence safety incident, or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(g)(h)A developer shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section no more than 30 days after making a covered model or covered model derivative available for commercial or public use for the first time. A developer need not submit a certification for a covered model derivative if the developer has already submitted a certification for the applicable covered model.
(h)(i)In fulfilling their obligations under this chapter, a developer shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
22604.(a)A person that operates a computing cluster shall implement written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(1)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(A)The identity of that prospective customer.
(B)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(C)The email address and telephonic contact information used to verify a prospective customers identity.
(2)Assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
(3)If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to paragraph (1) and conduct the assessment required pursuant to paragraph (2) prior to each utilization.
(4)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(5)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(6)Implement the capability to promptly enact a full shutdown of any resources being used to train or operate such customers administered models. models under the customers control.
(b)A person that operates a computing cluster shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General finds that a person is violating this chapter, the Attorney General may bring a civil action pursuant to this section.(b)Subject to subdivision (d), in a civil action under this section, the court may award any of the following:(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.(B)Relief pursuant to this paragraph shall be granted only in response to death or bodily harm to another human, harm to property, theft of property, or an imminent risk or threat to public safety.(2)Other relief as the court deems appropriate, such as monetary damages, including punitive damages, to persons aggrieved and an order for the full shutdown of a covered model.(3)A civil penalty in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.(c)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.(d)(1)For a violation that occurs before January 1, 2026, a court shall not do any of the following:(A)Order the deletion of a covered model and the weights utilized in that model.(B)Order the full shutdown of a covered model that does not present an imminent threat to public safety.(C)Award a civil penalty under paragraph (3) of subdivision (b).(2)For a violation that occurs before July 1, 2025, a court shall not award monetary damages to persons aggrieved.22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.(d)Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.(f)(1)Developers shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with the requirements of Section 22603 or has made false or materially misleading statements related to its safety and security protocol that includes, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer in response to the disclosure.(2)The disclosures and responses of the process required by this subdivision shall be maintained and shared with nonconflicted officers and directors of the company on a regular basis and not less than once per quarter.(g)As used in this section, employee has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:(1)Contractors or unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.(2)Corporate officers.22606.(a)The following are unlawful acts:
(1)For a developer to fail to comply with any of the requirements of Section 22603 or subdivision (a) of Section 22605.
(2)For a person who operates a computing cluster to fail to comply with the requirements of Section 22604 or subdivision (b) of Section 22605.
(3)For a developer to fail to comply with any of the requirements of Section 22607.
(b)The following parties may bring a civil action pursuant to subdivision (a):
(1)The Attorney General to enforce any provision of this chapter.
(2)The Labor Commissioner to enforce any provision of Section 22607 that would constitute a violation of the Labor Code.
(c)The parties listed in subdivision (b) are entitled to recover all of the following in addition to any civil penalties specified in this chapter:
(1)A civil penalty for a violation that occurs on or after January 1, 2026, in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(2)(A)Injunctive or declaratory relief, including, but not limited to, orders to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer.
(B)The court may only order relief under this paragraph for a covered model that has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.
(3)(A)Monetary damages.
(B)Punitive damages pursuant to subdivision (a) of Section 3294 of the Civil Code.
(4)Attorneys fees and costs.
(5)Any other relief that the court deems appropriate.
(d)(1)A provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of this chapter, or to shift that liability to any person or entity in exchange for their use or access of, or right to use or access, a developers products or services, including by means of a contract of adhesion, is void as a matter of public policy.
(2)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section to the maximum extent allowed by law if the court concludes that both of the following are true:
(A)The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
(B)As the result of the steps described in subparagraph (A), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(e)This section does not limit the application of other laws.
22607.(a)A developer of a covered model or a contractor or subcontractor of the developer shall not do either of the following:
(1)Prevent an employee from disclosing information to the Attorney General or the Labor Commissioner, including through terms and conditions of employment or seeking to enforce terms and conditions of employment, if the employee has reasonable cause to believe either of the following:
(A)The developer is out of compliance with the requirements of Section 22603.
(B)An artificial intelligence model, including a model that is not a covered model, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
(2)Retaliate against an employee for disclosing information to the Attorney General or Labor Commissioner, if the employee has reasonable cause to believe either subparagraph (A) or (B) of paragraph (1).
(3)Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 (commencing with Section 16600) of Division 7 or any other provision of state law.
(b)(1)An employee harmed by a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
(2)An employee of the Frontier Model Division may report any violation of this chapter by the Frontier Model Division to the State Auditor pursuant to the provisions of the California Whistleblower Protection Act (Article 3 (commencing with Section 8547) of Chapter 6.5 of Division 1 of Title 2 of the Government Code) which shall govern any such report.
(c)The Attorney General or Labor Commissioner may publicly release or provide, to the Frontier Model Division or the Governor, any complaint, or a summary of that complaint, pursuant to this section if they conclude that doing so will serve the public interest.
(d)A developer and any contractor or subcontractor of the developer shall provide a clear notice to all employees working on covered models of their rights and responsibilities under this section. A developer is presumed to be in compliance with the requirements of this subdivision if the developer does one of the following:
(1)At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice, and ensure that employees who work remotely periodically receive an equivalent notice.
(2)No less frequently than once every six months, provide written notice to all employees of their rights and responsibilities under this chapter and ensure that such notice is received and acknowledged by all those employees.
(e)(1)A developer and any contractor or subcontractor of the developer shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer has violated any provision of Section 22603 or any other law, or has made false or materially misleading statements related to its safety and security protocol, or failed to disclose known risks to employees, including, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer, contractor, or subcontractor in response to the disclosure.
(2)The disclosures and responses of the process required by this subdivision shall be maintained for a minimum of seven years from the date when the disclosure or response is created. Each disclosure and response shall be shared with officers and directors of the developer and any contractor or subcontractor of the developer whose acts or omissions are not implicated by the disclosure or response no less frequently than once per quarter.
(f)Nothing in this section shall be construed to limit protections provided to employees by Section 1102.5 of the Labor Code, Section 12964.5 of the Government Code, or other provisions of California law.
(g)As used in this section, the following definitions apply:
(1)Employee has the same meaning as defined in Section 1132.4 of the Labor Code and includes both of the following:
(A)Contractors or subcontractors, and unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
(B)Corporate officers.
(2)Contractor or subcontractor has the same meaning as in Section 1777.1 of the Labor Code.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section, critical harm has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)There is hereby established the Board of Frontier Models. The board shall be housed in the Government Operations Agency and shall be independent of the Department of Technology. The Governor may appoint an executive officer of the board, subject to Senate confirmation, who shall hold the office at the pleasure of the Governor. The executive officer shall be the administrative head of the board and shall exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged.
(c)Commencing January 1, 2026, the Board of Frontier Models shall be composed of five members, as follows:
(1)A member of the open-source community, appointed by the Governor, subject to Senate confirmation.
(2)A member of the artificial intelligence industry, appointed by the Governor, subject to Senate confirmation.
(3)A member of academia, appointed by the Governor, subject to Senate confirmation.
(4)A member appointed by the Speaker of the Assembly.
(5)A member appointed by the Senate Rules Committee.
(d)The Frontier Model Division is hereby created within the Government Operations Agency under the direct supervision of the Board of Frontier Models.
(e)The Frontier Model Division shall do all of the following:
(1)Annually review certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices necessary to prevent unreasonable risks of covered models and covered model derivatives causing or enabling critical harms, including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an accreditation process and relevant accreditation standards under which third-party auditors may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to their requirements under Section 22603 of the Business and Professions Code.(B)Issue guidance regarding best practices for conducting an audit pursuant to subdivision (e) of Section 22603 of the Business and Professions Code.
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(6)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(7)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that lack the ability to cause or enable critical harms.
(B)Advise the Legislature on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(8)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(9)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt. adopt consistent with its policies and procedures for the promulgation of jury instructions.
(B)In developing the proposed model jury instructions required by subparagraph (A), the Frontier Model Division shall consider and incorporate all of the following factors: factors into the proposal that it submits to the Judicial Council:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant risk of causing or enabling a critical harm.(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing or enabling a particular harm.(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.(i)All of the actions that a developer of a covered model must take pursuant to Section 22603 of the Business and Professions Code.
(ii)How any regulations of the Frontier Model Division should be incorporated into the proposed model jury instructions.
(iii)The rigor and quality of the safety and security protocol that a developer is required to implement while training and releasing its artificial intelligence model, and how to determine whether this safety and security protocol was inferior, comparable, or superior to the safety and security protocols of similarly situated developers.
(iv)The rigor and quality of the developers investigation, documentation, evaluation, and management of its models potential hazardous capabilities, and associated risks.
(10)(A)On or before January 1, 2027, and annually thereafter, issue regulations to update the definition of a covered model to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of causing or enabling critical harms. The updated definition shall contain both of the following:
(i)The initial compute threshold that an artificial intelligence model must exceed to be considered a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii)The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.
(B)In developing regulations pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i)The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or enable a critical harm.
(ii)Similar thresholds used in federal law, guidance, or regulations for the management of models with reasonable risks of causing or enabling critical harms.
(iii)Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
(11)Every 24 months after initial publication of guidance under paragraphs (3), (5), and (10), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(12)On and after January 1, 2026, annually publish the inflation-adjusted dollar amounts described in paragraph (3) of subdivision (g) and paragraph (2) of subdivision (e) of Section 22602 of the Business and Professions Code.
(f)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model intelligence means an engineered or machine-based system that, that varies in its level of autonomy and that can, for explicit or implicit objectives, infers, objectives infer from the input it receives, receives how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy. environments.
(c)Artificial intelligence safety incident means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
(1)A covered model autonomously engaging in behavior other than at the request of a user that materially increases the risk of a hazardous capability being used. user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a limited duty exemption. model.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a limited duty exemption. model.
(4)Unauthorized use of the hazardous capability of a covered model. model to cause or enable critical harm.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means either of the following:(1)Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.(2)Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities. (f)(e)(1)Covered model means an either of the following:
(A)Before January 1, 2027, covered model means either of the following:
(i)An artificial intelligence model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations, and the cost of that quantity of computing power would exceed which exceeds one hundred million dollars ($100,000,000) if when calculated using the average market prices prices of cloud compute as reasonably assessed by the developer at the time of training. at the start of training as reasonably assessed by the developer.
(ii)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than three times 10^25 integer or floating-point operations.
(B)(i)Except as provided in clause (ii), on and after January 1, 2027, covered model means any of the following:
(I)An artificial intelligence model trained using a quantity of computing power determined by the Frontier Model Division pursuant to Section 11547.6 of the Government Code, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
(II)An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division.
(ii)If the Frontier Model Division does not adopt a regulation governing subclauses (I) and (II) of clause (i) by January 1, 2027, the definition of covered model in subparagraph (A) continues to be in effect until the regulation is adopted.
(2)On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).(f)Covered model derivative means any of the following:
(1)An unmodified copy of a covered model.
(2)A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
(3)(A)(i)Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding three times 10^25 integer or floating point operations.
(ii)On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Frontier Model Division.
(B)If the Frontier Model Division does not adopt a regulation governing clause (ii) of subparagraph (A) by January 1, 2027, the quantity of computing power specified in clause (i) of subparagraph (A) shall continue to apply until the regulation is adopted.
(4)A copy of a covered model that has been combined with other software.
(g)(1)Critical harm means any of the following harms caused or enabled by a covered model or covered model derivative:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure, occurring either in a single incident or over multiple related incidents.
(C)Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model autonomously engaging in conduct that would constitute a serious or violent felony under the Penal Code if undertaken by a human with the requisite mental state.
(D)Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2)Critical harm does not include harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible.
(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:(A)A modified or unmodified copy of an artificial intelligence model.(B)A combination of an artificial intelligence model with other software.(2)Derivative model does not include either of the following: (A)An entirely independently trained artificial intelligence model.(B)An artificial intelligence model, including one combined with other software, that is fine-tuned using a quantity of computing power greater than 25 percent of the quantity of computing power, measured in integer or floating-point operations, used to train the original model.(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.(i)Developer means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, or by fine-tuning an existing covered model using sufficient quantity of computing power pursuant to subdivision (e).
(j)Fine-tuning means adjusting the model weights of a trained covered model by exposing it to additional data.
(l)(k)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)(1)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within the custody, control, or possession of a nonderivative model developer or a person that operates a computing cluster, including any computer or storage device remotely provided by agreement.(2)Full shutdown does not mean the cessation of operation of a covered model to which access was granted pursuant to a license that was not granted by the licensor on a discretionary basis and was not subject to separate negotiation between the parties.(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model that does not qualify for a limited duty exemption:(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human with the necessary mental state and causes either of the following:(i)Bodily harm to another human. (ii)The theft of, or harm to, property.(D)Other grave threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.(o)Limited duty exemption means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model, which applies if a developer can provide reasonable assurance that a covered model does not have a hazardous capability and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.(p)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.(l)Full shutdown means the cessation of operation of any of the following:
(1)The training of a covered model.
(2)A covered model.
(3)All covered model derivatives controlled by a developer.
(q)(m)Model weight means a numerical parameter established in an artificial intelligence model that is adjusted through training in an artificial intelligence model and that helps determine how input information impacts a models output. inputs are transformed into outputs.
(r)(n)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and that may be freely modified and redistributed.
(s)(o)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(t)Posttraining(p)Post-training modification means the modification of modifying the capabilities of an artificial intelligence a covered model after the completion of training by any means, including, but not limited to, initiating additional training, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)(q)Reasonable assurance does not mean full certainty or practical certainty.
(v)(r)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of derivative models. covered model derivatives.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model may determine whether the covered model qualifies for a limited duty exemption.(1)In making the determination authorized by this subdivision, a developer shall incorporate all applicable covered guidance.(2)A developer may determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and has an equal or lesser general capability than either of the following:(A)A noncovered model that manifestly lacks hazardous capabilities.(B)Another model that is the subject of a limited duty exemption.(3)Upon determining that a covered model qualifies for a limited duty exemption, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination.(4)A developer that makes a good faith error regarding a limited duty exemption shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).(b)Before initiating training of a covered model that is not a derivative model and is not the subject of a limited duty exemption, and until that covered model is the subject of a limited duty exemption, the developer of that covered model shall do all of the following:22603.(a)Before a developer initially trains a covered model, the developer shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse of, or unsafe modification post-training modifications of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model. shutdown.
(3)Implement all covered guidance.(4)(3)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.(ii)The safeguards enumerated in the protocol will be sufficient to prevent unreasonable risk of critical harms from the exercise of a hazardous capability in a covered model.(A)If a developer complies with the safety and security protocol, provides reasonable assurance that the developer will not produce a covered model or covered model derivative that poses an unreasonable risk of causing or enabling a critical harm.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to provide reasonable assurance that a covered model does not have a hazardous capability and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of both of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.(iv)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make posttraining modifications or create another covered model in a manner that may generate hazardous capabilities.(i)That a covered model does not pose an unreasonable risk of causing or enabling a critical harm.
(ii)That covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
(D)Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
(E)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make post-training modifications or create another covered model in a manner that may generate hazardous capabilities.
(v)(F)Provides sufficient detail for third parties to replicate the testing procedure.
(D)(G)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5). fulfill their obligations under this chapter.
(E)If applicable, describes(H)Describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d). this section.
(F)(I)Describes in detail the conditions that would require the execution of under which a developer would enact a full shutdown.
(G)(J)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.(5)(4)Ensure that the safety and security protocol is implemented as written, including, at a minimum, including by designating senior personnel to be responsible for ensuring implementation compliance by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate. third-party auditors.
(6)(5)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)(6)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)(7)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.(10)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.(c)(1)Upon completion of the training of a covered model that is not the subject of a limited duty exemption under subdivision (a) and is not a derivative model, the developer shall perform capability testing sufficient to determine if a limited duty exemption applies with respect to the covered model pursuant to its safety and security protocol.(2)Upon determining if a limited duty exemption applies with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:(A)The basis for the developers determination whether a limited duty exemption applies.(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, a developer of the nonderivative version of the covered model shall do all of the following:(1)Implement reasonable safeguards and requirements, informed by the training and testing process, to do all of the following:(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.(B)Prevent an individual from being able to use the model to create a derivative model that is used to cause a critical harm.(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.(2)(A)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.(B)If a developer provides access to the derivative model in a form that makes fine tuning possible, provide information to developers of that derivative model in a manner that will enable them to determine whether they have done a sufficient amount of fine tuning to meet the threshold described in subparagraph (B) of paragraph (2) of subdivision (i) of Section 22602.(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.(4)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.(8)Implement other reasonable measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or enabling critical harms.
(b)Before using a covered model or covered model derivative, or making a covered model or covered model derivative available for commercial or public use, the developer of a covered model shall do all of the following:
(1)Assess whether the covered model is reasonably capable of causing or enabling a critical harm.
(2)Implement reasonable safeguards to prevent the covered model and covered model derivatives from causing or enabling a critical harm.
(3)Ensure, to the extent reasonably possible, that the covered models actions and the actions of covered model derivatives, as well as critical harms resulting from their actions, can be accurately and reliably attributed to them.
(4)Beginning January 1, 2028, obtain a certificate of compliance from a third-party auditor who has been accredited pursuant to 11547.6 of the Government Code.
(c)A developer shall not use a covered model commercially or publicly, or make a covered model or a covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.
(e)(d)A developer of a nonderivative covered model shall periodically annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards. section.
(f)(e)(1)A developer of a nonderivative covered model that is not the subject of a limited duty exemption shall annually submit to the Frontier Model Division an annual a certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. This paragraph applies as long as the covered model or any covered model derivatives controlled by the developer remain in commercial or public use, or remain available for commercial or public use.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities critical harms that the covered model possesses or may reasonably possess or covered model derivatives may reasonably cause or enable, and the outcome of capability testing the assessment required by subdivision (c). (b).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities. the covered model or covered model derivatives from causing or enabling critical harms.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.(C)A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the certification, and the contact information of any third parties relied upon to validate compliance.
(D)Beginning January 1, 2028, a certificate of compliance from an accredited third-party auditor.
(g)(1)(f)A developer of a nonderivative covered model shall report each artificial intelligence safety incident affecting that the covered model and model, or any derivative version of that covered model within the custody, control, or possession of derivatives controlled by the developer, as described in subdivision (m) of Section 22602, to the Frontier Model Division in a manner prescribed by the Frontier Model Division. within 72 hours of the developer learning of the artificial intelligence safety incident, or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(2)The report required by this subdivision shall be made not later than 72 hours after the developer learns that an artificial intelligence safety incident has occurred, or the developer learns facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.(h)(1)(A)Reliance on an unreasonable limited duty exemption does not relieve a developer of its obligations under this section.(B)A determination that a covered model qualifies for a limited duty exemption that results from a good faith error reported pursuant to paragraph (4) of subdivision (a) is not an unreasonable limited duty exemption.(2)A limited duty exemption is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.(B)By the National Institute of Standards and Technology, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.(g)A developer shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section no more than 30 days after making a covered model or covered model derivative available for commercial or public use for the first time. A developer need not submit a certification for a covered model derivative if the developer has already submitted a certification for the applicable covered model.
(h)In fulfilling their obligations under this chapter, a developer shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
22604.(a)A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)(1)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)(A)The identity of that prospective customer.
(2)(B)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)(C)The email address and telephonic contact information used to verify a prospective customers identity.
(b)(2)Assess whether a prospective customer intends to utilize the computing cluster to deploy train a covered model.
(c)Annually(3)If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to subdivision (a) paragraph (1) and conduct the assessment required pursuant to subdivision (b). paragraph (2) prior to each utilization.
(4)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(d)(5)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)(6)Implement the capability to promptly enact a full shutdown in the event of an emergency. of any resources being used to train or operate such customers administered models.
(f)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.(b)A person that operates a computing cluster shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General finds that a person is violating this chapter, the Attorney General may bring a civil action pursuant to this section.
(b)Subject to subdivision (d), in a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to death or bodily harm to another human, harm to property, theft of property, or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, such as monetary damages, including punitive damages, to persons aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(c)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(d)(1)For a violation that occurs before January 1, 2026, a court shall not do any of the following:
(A)Order the deletion of a covered model and the weights utilized in that model.
(B)Order the full shutdown of a covered model that does not present an imminent threat to public safety.
(C)Award a civil penalty under paragraph (3) of subdivision (b).
(2)For a violation that occurs before July 1, 2025, a court shall not award monetary damages to persons aggrieved.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
(f)(1)Developers shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with the requirements of Section 22603 or has made false or materially misleading statements related to its safety and security protocol that includes, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer in response to the disclosure.
(2)The disclosures and responses of the process required by this subdivision shall be maintained and shared with nonconflicted officers and directors of the company on a regular basis and not less than once per quarter.
(g)As used in this section, employee has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
(1)Contractors or unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
(2)Corporate officers.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability section, critical harm has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Limited duty exemption has the same meaning as defined in Section 22602 of the Business and Professions Code.(b)There is hereby established the Board of Frontier Models. The board shall be housed in the Government Operations Agency and shall be independent of the Department of Technology. The Governor may appoint an executive officer of the board, subject to Senate confirmation, who shall hold the office at the pleasure of the Governor. The executive officer shall be the administrative head of the board and shall exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged.
(c)Commencing January 1, 2026, the Board of Frontier Models shall be composed of five members, as follows:
(1)A member of the open-source community, appointed by the Governor, subject to Senate confirmation.
(2)A member of the artificial intelligence industry, appointed by the Governor, subject to Senate confirmation.
(3)A member of academia, appointed by the Governor, subject to Senate confirmation.
(4)A member appointed by the Speaker of the Assembly.
(5)A member appointed by the Senate Rules Committee.
(b)(d)The Frontier Model Division is hereby created within the Department of Technology. Government Operations Agency under the direct supervision of the Board of Frontier Models.
(c)(e)The Frontier Model Division shall do all of the following:
(1)Annually review certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices necessary to prevent unreasonable risks from of covered models with hazardous capabilities and covered model derivatives causing or enabling critical harms, including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an optional accreditation process and relevant accreditation standards under which third parties third-party auditors may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A). their requirements under Section 22603 of the Business and Professions Code.
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(6)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(7)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities. lack the ability to cause or enable critical harms.
(B)Advise the Frontier Model Division Legislature on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(8)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(9)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities. risk of causing or enabling a critical harm.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing or enabling a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
(10)(A)On or before July 1, 2026, issue guidance regarding both of the following:(i)Information relevant to determining whether an artificial intelligence model is a covered model, as defined in Section 22602 of the Business and Professions Code.(ii)Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code.(10)(A)On or before January 1, 2027, and annually thereafter, issue regulations to update the definition of a covered model to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of causing or enabling critical harms. The updated definition shall contain both of the following:
(i)The initial compute threshold that an artificial intelligence model must exceed to be considered a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii)The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.
(B)In developing guidance regulations pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i)The quantity of computing power used to train covered models that have been identified as having hazardous capabilities or not having hazardous capabilities when accounting for a reasonable margin for safety. being reasonably likely to cause or enable a critical harm.
(ii)Similar thresholds used in federal law law, guidance, or regulations for the management of hazardous capabilities. models with reasonable risks of causing or enabling critical harms.
(iii)Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
(11)At least every Every 24 months after initial publication of guidance under paragraphs (3), (5), and (10), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(12)On and after January 1, 2026, annually publish the inflation-adjusted dollar amounts described in paragraph (3) of subdivision (n) (g) and paragraph (2) of subdivision (f) (e) of Section 22602 of the Business and Professions Code.
(d)(f)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in behavior other than at the request of a user that materially increases the risk of a hazardous capability being used.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a limited duty exemption.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a limited duty exemption.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means either of the following:
(1)Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(2)Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(f)(1)Covered model means an artificial intelligence model that meets either of the following criteria: was trained using a quantity of computing power greater than 10^26 integer or floating-point operations, and the cost of that quantity of computing power would exceed one hundred million dollars ($100,000,000) if calculated using average market prices of cloud compute as reasonably assessed by the developer at the time of training.
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.(2)The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.(2)On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model. either of the following:
(A)An entirely independently trained artificial intelligence model.
(B)An artificial intelligence model, including one combined with other software, that is fine-tuned using a quantity of computing power greater than 25 percent of the quantity of computing power, measured in integer or floating-point operations, used to train the original model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)(1)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within the custody, control, or possession of a person, nonderivative model developer or a person that operates a computing cluster, including any computer or storage device remotely provided by agreement.
(2)Full shutdown does not mean the cessation of operation of a covered model to which access was granted pursuant to a license that was not granted by the licensor on a discretionary basis and was not subject to separate negotiation between the parties.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model: model that does not qualify for a limited duty exemption:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human. human with the necessary mental state and causes either of the following:
(i)Bodily harm to another human.
(ii)The theft of, or harm to, property.
(D)Other grave threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(3)On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(o)Limited duty exemption means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that model, which applies if a developer can reasonably exclude the possibility provide reasonable assurance that a covered model has does not have a hazardous capability or may and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(p)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(q)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(r)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(s)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Reasonable assurance does not mean full certainty or practical certainty.
(u)(v)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model may determine whether the covered model qualifies for a limited duty exemption.
(1)In making the determination authorized by this subdivision, a developer shall incorporate all applicable covered guidance.
(2)A developer may determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and does not have greater has an equal or lesser general capability than either of the following:
(A)A noncovered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a limited duty exemption.
(3)Upon determining that a covered model qualifies for a limited duty exemption, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination.
(4)A developer that makes a good faith error regarding a limited duty exemption shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b)Before initiating training of a covered model that is not a derivative model and is not the subject of a limited duty exemption, and until that covered model is the subject of a limited duty exemption, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy protocol will be sufficient to prevent unreasonable risk of critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility provide reasonable assurance that a covered model has does not have a hazardous capability or may and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make posttraining modifications or create another covered model in a manner that may generate hazardous capabilities.
(v)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(10)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(c)(1)Upon completion of the training of a covered model that is not the subject of a limited duty exemption under subdivision (a) and is not a derivative model, the developer shall perform capability testing sufficient to determine if a limited duty exemption applies with respect to the covered model pursuant to its safety and security protocol.
(2)Upon determining if a limited duty exemption applies with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers determination whether a limited duty exemption applies.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements requirements, informed by the training and testing process, to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was is used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)(A)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(B)If a developer provides access to the derivative model in a form that makes fine tuning possible, provide information to developers of that derivative model in a manner that will enable them to determine whether they have done a sufficient amount of fine tuning to meet the threshold described in subparagraph (B) of paragraph (2) of subdivision (i) of Section 22602.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(4)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(e)A developer of a nonderivative covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a limited duty exemption shall submit to the Frontier Model Division an annual certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)(1)A developer of a nonderivative covered model shall report each artificial intelligence safety incident affecting a that covered model and any derivative version of that covered model within the custody, control, or possession of the developer, as described in subdivision (m) of Section 22602, to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(2)The report required by this subdivision shall be made not later than 72 hours after the developer learns that an artificial intelligence safety incident has occurred, or the developer learns facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)(A)Reliance on an unreasonable limited duty exemption does not relieve a developer of its obligations under this section.
(B)A determination that a covered model qualifies for a limited duty exemption that results from a good faith error reported pursuant to paragraph (4) of subdivision (a) is not an unreasonable limited duty exemption.
(2)A limited duty exemption is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the National Institute of Standards and Technology, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
(f)Retain a customers Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General finds that a person is violating this chapter, the Attorney General may bring a civil action pursuant to this section.
(b)Subject to subdivision (d), in a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to death or bodily harm to another human, harm to property, theft of property, or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, such as monetary damages, including punitive damages, to persons aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model that value for any subsequent violation.
(c)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(d)(1)For a violation that occurs before January 1, 2026, a court shall not do any of the following:
(A)Order the deletion of a covered model and the weights utilized in that model.
(B)Order the full shutdown of a covered model that does not present an imminent threat to public safety.
(C)Award a civil penalty under paragraph (3) of subdivision (b).
(2)For a violation that occurs before July 1, 2025, a court shall not award monetary damages to persons aggrieved.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
(f)(1)Developers shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with the requirements of Section 22603 or has made false or materially misleading statements related to its safety and security protocol that includes, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer in response to the disclosure.
(2)The disclosures and responses of the process required by this subdivision shall be maintained and shared with nonconflicted officers and directors of the company on a regular basis and not less than once per quarter.
(g)As used in this section, employee has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
(1)Contractors or unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
(2)Corporate officers.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Limited duty exemption has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual Annually review certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices necessary to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an optional accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(6)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(7)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(8)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(9)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
(10)(A)On or before July 1, 2026, issue guidance regarding both of the following:
(i)Technical thresholds and benchmarks Information relevant to determining whether an artificial intelligence model is a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii)Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code.
(B)In developing guidance pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i)The quantity of computing power used to train covered models that have been identified as having hazardous capabilities or not having hazardous capabilities when accounting for a reasonable margin for safety.
(ii)Similar thresholds used in federal law or regulations for the management of hazardous capabilities.
(11)At least every 24 months after initial publication of guidance under paragraphs (3), (5), and (10), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(12)On and after January 1, 2026, annually publish the inflation-adjusted dollar amounts described in paragraph (3) of subdivision (n) and paragraph (2) of subdivision (f) of Section 22602 of the Business and Professions Code.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in behavior other than at the request of a user that materially increases the risk of a hazardous capability being used.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a limited duty exemption.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a limited duty exemption.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means either of the following:
(1)Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(2)Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(f)Covered model means an artificial intelligence model that meets either of the following criteria:
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
(2)The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(o)Limited duty exemption means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(p)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(q)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(r)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(s)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model may determine whether the covered model qualifies for a limited duty exemption.
(1)In making the determination authorized by this subdivision, a developer shall incorporate all applicable covered guidance.
(2)A developer may determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and does not have greater general capability than either of the following:
(A)A noncovered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a limited duty exemption.
(3)Upon determining that a covered model qualifies for a limited duty exemption, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination.
(4)A developer that makes a good faith error regarding a limited duty exemption shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b)Before initiating training of a covered model that is not a derivative model and is not the subject of a limited duty exemption, and until that covered model is the subject of a limited duty exemption, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy will be sufficient to prevent critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make posttraining modifications or create another covered model in a manner that may generate hazardous capabilities.
(v)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(10)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(c)(1)Upon completion of the training of a covered model that is not the subject of a limited duty exemption under subdivision (a) and is not a derivative model, the developer shall perform capability testing sufficient to determine if a limited duty exemption applies with respect to the covered model pursuant to its safety and security protocol.
(2)Upon determining if a limited duty exemption applies with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers determination whether a limited duty exemption applies.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(4)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(e)A developer of a covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a limited duty exemption shall submit to the Frontier Model Division an annual certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)A developer shall report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)(A)Reliance on an unreasonable limited duty exemption does not relieve a developer of its obligations under this section.
(B)A determination that a covered model qualifies for a limited duty exemption that results from a good faith error reported pursuant to paragraph (4) of subdivision (a) is not an unreasonable limited duty exemption.
(2)A limited duty exemption is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the National Institute of Standards and Technology, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General finds that a person is violating this chapter, the Attorney General may bring a civil action pursuant to this section.
(b)Subject to subdivision (d), in a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to harm or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, such as monetary damages, including punitive damages, to persons aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation.
(c)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(d)(1)For a violation that occurs before January 1, 2026, a court shall not do any of the following:
(A)Order the deletion of a covered model and the weights utilized in that model.
(B)Order the full shutdown of a covered model that does not present an imminent threat to public safety.
(C)Award a civil penalty under paragraph (3) of subdivision (b).
(2)For a violation that occurs before July 1, 2025, a court shall not award monetary damages to persons aggrieved.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
(f)(1)Developers shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with the requirements of Section 22603 or has made false or materially misleading statements related to its safety and security protocol that includes, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer in response to the disclosure.
(2)The disclosures and responses of the process required by this subdivision shall be maintained and shared with nonconflicted officers and directors of the company on a regular basis and not less than once per quarter.
(g)As used in this section, employee has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
(1)Contractors or unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
(2)Corporate officers.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Limited duty exemption has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices necessary to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an optional accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.(6)(5)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(7)(6)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(8)(7)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(9)Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.(10)Monitor relevant developments relating to the safety risks associated with the development of artificial intelligence models and the functioning of markets for artificial intelligence models.(11)(8)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(12)(9)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
(13)(10)(A)On or before July 1, 2026, issue guidance regarding both of the following:
(i)Technical thresholds and benchmarks relevant to determining whether an artificial intelligence model is a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii)Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code.
(B)In developing guidance pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i)The quantity of computing power used to train covered models that have been identified as having hazardous capabilities or not having hazardous capabilities when accounting for a reasonable margin for safety.
(ii)Similar thresholds used in federal law or regulations for the management of hazardous capabilities.
(14)(11)At least every 24 months after initial publication of guidance under paragraphs (3) and (13), (3), (5), and (10), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in behavior other than at the request of a user that materially increases the risk of a hazardous capability being used.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a limited duty exemption.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a limited duty exemption.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means either of the following:
(1)Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(2)Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(f)Covered model means an artificial intelligence model that meets either of the following criteria:
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
(2)The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(o)Limited duty exemption means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(p)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(q)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(r)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(s)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model may determine whether the covered model qualifies for a limited duty exemption.
(1)In making the determination authorized by this subdivision, a developer shall incorporate all applicable covered guidance.
(2)A developer may determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and does not have greater general capability than either of the following:
(A)A noncovered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a limited duty exemption.
(3)Upon determining that a covered model qualifies for a limited duty exemption, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination.
(4)A developer that makes a good faith error regarding a limited duty exemption shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b)Before initiating training of a covered model that is not a derivative model and is not the subject of a limited duty exemption, and until that covered model is the subject of a limited duty exemption, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy will be sufficient to prevent critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make posttraining modifications or create another covered model in a manner that may generate hazardous capabilities.
(v)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(10)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(c)(1)Upon completion of the training of a covered model that is not the subject of a limited duty exemption under subdivision (a) and is not a derivative model, the developer shall perform capability testing sufficient to determine whether the developer can make a positive safety determination if a limited duty exemption applies with respect to the covered model pursuant to its safety and security protocol.
(2)Upon making determining if a limited duty exemption applies with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division Division, under penalty of perjury, a certification under penalty of perjury of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers determination that the covered model qualifies for whether a limited duty exemption. exemption applies.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(4)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(e)A developer of a covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a limited duty exemption shall submit to the Frontier Model Division an annual certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)A developer shall report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)(A)Reliance on an unreasonable limited duty exemption does not relieve a developer of its obligations under this section.
(B)A determination that a covered model qualifies for a limited duty exemption that results from a good faith error reported pursuant to paragraph (4) of subdivision (a) is not an unreasonable limited duty exemption.
(2)A limited duty exemption is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the National Institute of Standards and Technology, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General finds that a person is violating this chapter, the Attorney General may bring a civil action pursuant to this section.
(b)In Subject to subdivision (d), in a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to harm or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, including such as monetary damages, including punitive damages, to persons aggrieved, punitive damages, aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation.
(c)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(d)(1)For a violation that occurs before January 1, 2026, a court shall not do any of the following:
(A)Order the deletion of a covered model and the weights utilized in that model.
(B)Order the full shutdown of a covered model that does not present an imminent threat to public safety.
(C)Award a civil penalty under paragraph (3) of subdivision (b).
(2)For a violation that occurs before July 1, 2025, a court shall not award monetary damages to persons aggrieved.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
(f)(1)Developers shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with the requirements of Section 22603 or has made false or materially misleading statements related to its safety and security protocol that includes, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer in response to the disclosure.
(2)The disclosures and responses of the process required by this subdivision shall be maintained and shared with nonconflicted officers and directors of the company on a regular basis and not less than once per quarter.
(g)As used in this section, employee has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
(1)Contractors or unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
(2)Corporate officers.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Limited duty exemption has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices necessary to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an optional accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
(6)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(7)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(8)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(9)Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
(10)Monitor relevant developments relating to the safety risks associated with the development of artificial intelligence models and the functioning of markets for artificial intelligence models.
(11)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(12)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
(13)(A)On or before July 1, 2026, issue guidance regarding both of the following:
(i)Technical thresholds and benchmarks relevant to determining whether an artificial intelligence model is a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii)Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code.
(B)In developing guidance pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i)The quantity of computing power used to train covered models that have been identified as having hazardous capabilities or not having hazardous capabilities when accounting for a reasonable margin for safety.
(ii)Similar thresholds used in federal law or regulations for the management of hazardous capabilities.
(14)At least every 24 months after initial publication of guidance under paragraphs (3) and (13), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in a sustained sequence of unsafe behavior other than at the request of a user. user that materially increases the risk of a hazardous capability being used.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a positive safety determination. limited duty exemption.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a positive safety determination. limited duty exemption.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means any either of the following:
(1)Applicable guidance Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division. Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(2)Industry best practices, including relevant safety practices, precautions, or testing procedures undertaken by developers of comparable models, and any safety standards or best practices commonly or generally recognized by relevant experts in academia or the nonprofit sector. models that are relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(3)Applicable safety-enhancing standards set by standards setting organizations.(f)Covered model means an artificial intelligence model that meets either of the following criteria:
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.
(2)The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(o)Limited duty exemption means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(o)(p)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(p)(q)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(q)(r)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(r)(s)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(s)Positive safety determination means a determination, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model shall may determine whether it can make the covered model qualifies for a positive safety determination with respect to the covered model. limited duty exemption.
(1)In making the determination required authorized by this subdivision, a developer shall incorporate all applicable covered guidance.
(2)A developer may make a positive safety determination determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and does not have greater general capability than either of the following:
(A)A non-covered noncovered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a positive safety determination. limited duty exemption.
(3)Upon making a positive safety determination, determining that a covered model qualifies for a limited duty exemption, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that conclusion. determination.
(4)A developer that makes a good faith error regarding a positive safety determination limited duty exemption shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b)Before initiating training of a covered model that is not a derivative model that and is not the subject of a positive safety determination, limited duty exemption, and until that covered model is the subject of a positive safety determination, limited duty exemption, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy will be sufficient to prevent critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make posttraining modifications or create another covered model in a manner that may generate hazardous capabilities.
(iv)(v)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(10)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(c)(1)Upon completion of the training of a covered model that is not the subject of a positive safety determination limited duty exemption and is not a derivative model, the developer shall perform capability testing sufficient to determine whether the developer can make a positive safety determination with respect to the covered model pursuant to its safety and security protocol.
(2)Upon making a positive safety determination limited duty exemption with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers positive safety determination. determination that the covered model qualifies for a limited duty exemption.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a positive safety determination, limited duty exemption, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(4)Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(e)A developer of a covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a positive safety determination limited duty exemption shall submit to the Frontier Model Division an annual certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)A developer shall report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)(A)Reliance on an unreasonable positive safety determination limited duty exemption does not relieve a developer of its obligations under this section.
(B)A determination that a covered model qualifies for a limited duty exemption that results from a good faith error reported pursuant to paragraph (4) of subdivision (a) is not an unreasonable limited duty exemption.
(2)A positive safety determination limited duty exemption is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the United States Artificial Intelligence Safety Institute, National Institute of Standards and Technology, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General has reasonable cause to believe finds that a person is violating this chapter, the Attorney General shall commence may bring a civil action in a court of competent jurisdiction. pursuant to this section.
(b)In a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to harm or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, including monetary damages damages, including punitive damages, to persons aggrieved aggrieved, punitive damages, and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation.
(c)In the apportionment of penalties assessed pursuant to this section, defendants shall be jointly and severally liable.(d)(c)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of this section subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
(f)(1)Developers shall provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with the requirements of Section 22603 or has made false or materially misleading statements related to its safety and security protocol that includes, at a minimum, a monthly update to the disclosing employee regarding the status of the employees disclosure and the actions taken by the developer in response to the disclosure.
(2)The disclosures and responses of the process required by this subdivision shall be maintained and shared with nonconflicted officers and directors of the company on a regular basis and not less than once per quarter.
(g)As used in this section, employee has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
(1)Contractors or unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
(2)Corporate officers.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Positive safety determination Limited duty exemption has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices sufficient necessary to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific components of or requirements on under the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an optional accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
(6)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(7)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(8)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(9)Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
(10)Monitor relevant developments relating to the safety risks associated with the development of artificial intelligence models and the functioning of markets for artificial intelligence models.
(11)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(12)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions brought by individuals injured by a hazardous capability of a covered model. involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state of the art, state-of-the-art, relevant risks that its model might pose.
(13)(A)On or before July 1, 2026, issue guidance regarding both of the following:
(i)Technical thresholds and benchmarks relevant to determining whether an artificial intelligence model is a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii)Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code.
(B)In developing guidance pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i)The quantity of computing power used to train covered models that have been identified as having hazardous capabilities or not having hazardous capabilities when accounting for a reasonable margin for safety.
(ii)Similar thresholds used in federal law or regulations for the management of hazardous capabilities.
(14)At least every 24 months after initial publication of guidance under paragraphs (3) and (13), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means a machine-based system designed to operate with varying levels of autonomy that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in a sustained sequence of unsafe behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a positive safety determination.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a positive safety determination.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means any of the following:
(1)Applicable guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division.
(2)Industry best practices, including relevant safety practices, precautions, or testing procedures undertaken by developers of comparable models, and any safety standards or best practices commonly or generally recognized by relevant experts in academia or the nonprofit sector.
(3)Applicable safety-enhancing standards set by standards setting organizations.
(f)Covered model means an artificial intelligence model that meets either of the following criteria:
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024. operations.
(2)The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(o)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(p)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(q)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(r)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(s)Positive safety determination means a determination, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model shall determine whether it can make a positive safety determination with respect to the covered model.
(1)In making the determination required by this subdivision, a developer shall incorporate all covered guidance.
(2)A developer may make a positive safety determination if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and does not have greater general capability than either of the following:
(A)A non-covered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a positive safety determination.
(3)Upon making a positive safety determination, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that conclusion.
(4)A developer that makes a good faith error regarding a positive safety determination shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b)Before initiating training of a covered model that is not a derivative model that is not the subject of a positive safety determination, and until that covered model is the subject of a positive safety determination, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy will be sufficient to prevent critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(c)(1)Upon completion of the training of a covered model that is not the subject of a positive safety determination and is not a derivative model, the developer shall perform capability testing sufficient to determine whether the developer can make a positive safety determination with respect to the covered model pursuant to its safety and security protocol.
(2)Upon making a positive safety determination with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers positive safety determination.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a positive safety determination, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(e)A developer of a covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a positive safety determination shall submit to the Frontier Model Division an annual certification of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)A developer shall report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)Reliance on an unreasonable positive safety determination does not relieve a developer of its obligations under this section.
(2)A positive safety determination is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the United States Artificial Intelligence Safety Institute, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General has reasonable cause to believe that a person is violating this chapter, the Attorney General shall commence a civil action in a court of competent jurisdiction.
(b)In a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to harm or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, including monetary damages to persons aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation.
(c)In the apportionment of penalties assessed pursuant to this section, defendants shall be jointly and severally liable.
(d)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of this section pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Positive safety determination has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices sufficient to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific requirements on the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
(6)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(7)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(8)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(9)Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
(10)Monitor relevant developments relating to the safety risks associated with the development of artificial intelligence models and the functioning of markets for artificial intelligence models.
(11)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(12)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions brought by individuals injured by a hazardous capability of a covered model.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state of the art, relevant risks that its model might pose.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Systems Models Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Systems Models22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means a machine-based system that can make predictions, recommendations, or decisions influencing real or virtual environments and can use model inference to formulate options for information or action. designed to operate with varying levels of autonomy that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in a sustained sequence of unsafe behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model. model that is not the subject of a positive safety determination.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model, designed to limit access to a hazardous capability of a covered model. model that is not the subject of a positive safety determination.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e)Covered guidance means any of the following:
(1)Applicable guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division.
(2)Industry best practices, including relevant safety practices, precautions, or testing procedures undertaken by developers of comparable models, and any safety standards or best practices commonly or generally recognized by relevant experts in academia or the nonprofit sector.
(3)Applicable safety-enhancing standards set by standards setting organizations.
(f)Covered model means an artificial intelligence model that meets either of the following criteria:
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024, or a model that could reasonably be expected to have similar performance on benchmarks commonly used to quantify the performance of state-of-the-art foundation models, as determined by industry best practices and relevant standard setting organizations. 2024.
(2)The artificial intelligence model has capability below the relevant threshold on a specific benchmark but is of otherwise similar general capability.(2)The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model that has been previously trained after it has finished its initial training by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(o)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(p)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(q)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(r)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(s)Positive safety determination means a determination, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model shall determine whether it can make a positive safety determination with respect to the covered model.
(1)In making the determination required by this subdivision, a developer shall incorporate all covered guidance.
(2)A developer may make a positive safety determination if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and does not have greater general capability than either of the following:
(A)A non-covered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a positive safety determination.
(3)Upon making a positive safety determination, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that conclusion.
(4)A developer that makes a good faith error regarding a positive safety determination shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b)Before initiating training of a covered model that is not a derivative model that is not the subject of a positive safety determination, and until that covered model is the subject of a positive safety determination, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy will be sufficient to prevent critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(c)(1)Upon completion of the training of a covered model that is not the subject of a positive safety determination and is not a derivative model, the developer shall perform capability testing sufficient to determine whether the developer can make a positive safety determination with respect to the covered model pursuant to its safety and security protocol.
(2)Upon making a positive safety determination with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers positive safety determination.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a positive safety determination, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(e)A developer of a covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a positive safety determination shall submit to the Frontier Model Division an annual certification of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)A developer shall report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)Reliance on an unreasonable positive safety determination does not relieve a developer of its obligations under this section.
(2)A positive safety determination is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the United States Artificial Intelligence Safety Institute, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)(1)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(2)A person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
22606.(a)If the Attorney General has reasonable cause to believe that a person is violating this chapter, the Attorney General shall commence a civil action in a court of competent jurisdiction.
(b)In a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to harm or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, including monetary damages to persons aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation.
(c)In the apportionment of penalties assessed pursuant to this section, defendants shall be jointly and severally liable.
(d)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of this section pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
22608.The duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Positive safety determination has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices sufficient to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific requirements on the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
(6)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(7)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(8)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(9)Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
(10)Monitor relevant developments relating to the safety risks associated with the development of artificial intelligence models and the functioning of markets for artificial intelligence models.
(11)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(12)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions brought by individuals injured by a hazardous capability of a covered model.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the mandatory safety policies and security protocols of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state of the art, relevant risks that its model might pose.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, public universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law.SEC. 9.SEC. 8.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
This act shall be known, and may be cited, as the Safe and Secure Innovation for Frontier Artificial Intelligence Systems Act.
SEC. 2.The Legislature finds and declares all of the following:
(a)California is leading the world in artificial intelligence innovation and research, through companies large and small, as well as through our remarkable public and private universities.
(b)Artificial intelligence, including new advances in generative artificial intelligence, has the potential to catalyze innovation and the rapid development of a wide range of benefits for Californians and the California economy, including advances in medicine, wildfire forecasting and prevention, and climate science, and to push the bounds of human creativity and capacity.
(c)If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.
(d)The state government has an essential role to play in ensuring that California recognizes the benefits of this technology while avoiding the most severe risks, as well as to ensure that artificial intelligence innovation and access to compute is accessible to academic researchers and startups, in addition to large companies.
SEC. 3.Chapter 22.6 (commencing with Section 22602) is added to Division 8 of the Business and Professions Code, to read:22.6.Safe and Secure Innovation for Frontier Artificial Intelligence Systems22602.As used in this chapter:
(a)Advanced persistent threat means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b)Artificial intelligence model means a machine-based system that can make predictions, recommendations, or decisions influencing real or virtual environments and can use model inference to formulate options for information or action.
(c)Artificial intelligence safety incident means any of the following:
(1)A covered model autonomously engaging in a sustained sequence of unsafe behavior other than at the request of a user.
(2)Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model.
(3)The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model, designed to limit access to a hazardous capability of a covered model.
(4)Unauthorized use of the hazardous capability of a covered model.
(d)Computing cluster means a set of machines transitively connected by data center networking of over 100 gigabits that has a theoretical maximum computing capacity of 10^20 integer or floating-point operations per second for training artificial intelligence.
(e)Covered guidance means any of the following:
(1)Applicable guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division.
(2)Industry best practices, including relevant safety practices, precautions, or testing procedures undertaken by developers of comparable models, and any safety standards or best practices commonly or generally recognized by relevant experts in academia or the nonprofit sector.
(3)Applicable safety-enhancing standards set by standards setting organizations.
(f)Covered model means an artificial intelligence model that meets either of the following criteria:
(1)The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024, or a model that could reasonably be expected to have similar performance on benchmarks commonly used to quantify the performance of state-of-the-art foundation models, as determined by industry best practices and relevant standard setting organizations.
(2)The artificial intelligence model has capability below the relevant threshold on a specific benchmark but is of otherwise similar general capability.
(g)Critical harm means a harm listed in paragraph (1) of subdivision (n).
(h)Critical infrastructure means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(i)(1)Derivative model means an artificial intelligence model that is a derivative of another artificial intelligence model, including either of the following:
(A)A modified or unmodified copy of an artificial intelligence model.
(B)A combination of an artificial intelligence model with other software.
(2)Derivative model does not include an entirely independently trained artificial intelligence model.
(j)(1)Developer means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2)Developer does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k)Fine tuning means the adjustment of the model weights of an artificial intelligence model that has been previously trained by training the model with new data.
(l)Frontier Model Division means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m)Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
(n)(1)Hazardous capability means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
(A)The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B)At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
(C)At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
(D)Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2)Hazardous capability includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(o)Machine-learning operations platform means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(p)Model weight means a numerical parameter established through training in an artificial intelligence model that helps determine how input information impacts a models output.
(q)Open-source artificial intelligence model means an artificial intelligence model that is made freely available and may be freely modified and redistributed.
(r)Person means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(s)Positive safety determination means a determination, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(t)Posttraining modification means the modification of the capabilities of an artificial intelligence model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(u)Safety and security protocol means documented technical and organizational protocols that meet both of the following criteria:
(1)The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models.
(2)The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developers covered model.
22603.(a)Before initiating training of a covered model that is not a derivative model, a developer of that covered model shall determine whether it can make a positive safety determination with respect to the covered model.
(1)In making the determination required by this subdivision, a developer shall incorporate all covered guidance.
(2)A developer may make a positive safety determination if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 than either of the following:
(A)A non-covered model that manifestly lacks hazardous capabilities.
(B)Another model that is the subject of a positive safety determination.
(3)Upon making a positive safety determination, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that conclusion.
(b)Before initiating training of a covered model that is not a derivative model that is not the subject of a positive safety determination, and until that covered model is the subject of a positive safety determination, the developer of that covered model shall do all of the following:
(1)Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developers custody, that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2)Implement the capability to promptly enact a full shutdown of the covered model.
(3)Implement all covered guidance.
(4)Implement a written and separate safety and security protocol that does all of the following:
(A)Provides reasonable assurance that if a developer complies with its safety and security protocol, either of the following will apply:
(i)The developer will not produce a covered model with a hazardous capability or enable the production of a derivative model with a hazardous capability.
(ii)The safeguards enumerated in the policy will be sufficient to prevent critical harms from the exercise of a hazardous capability in a covered model.
(B)States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C)Identifies specific tests and test results that would be sufficient to reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:
(i)Describes in detail how the testing procedure incorporates fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(ii)Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii)Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv)Provides sufficient detail for third parties to replicate the testing procedure.
(D)Describes in detail how the developer will meet requirements listed under paragraphs (1), (2), (3), and (5).
(E)If applicable, describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(F)Describes in detail the conditions that would require the execution of a full shutdown.
(G)Describes in detail the procedure by which the safety and security protocol may be modified.
(H)Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5)Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel responsible for ensuring implementation by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriate.
(6)Provide a copy of the safety and security protocol to the Frontier Model Division.
(7)Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(8)If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9)Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
(c)(1)Upon completion of the training of a covered model that is not the subject of a positive safety determination and is not a derivative model, the developer shall perform capability testing sufficient to determine whether the developer can make a positive safety determination with respect to the covered model pursuant to its safety and security protocol.
(2)Upon making a positive safety determination with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A)The basis for the developers positive safety determination.
(B)The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d)Before initiating the commercial, public, or widespread use of a covered model that is not subject to a positive safety determination, a developer of the nonderivative version of the covered model shall do all of the following:
(1)Implement reasonable safeguards and requirements to do all of the following:
(A)Prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm.
(B)Prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm.
(C)Ensure, to the extent reasonably possible, that the covered models actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
(2)Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(3)Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(e)A developer of a covered model shall periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(f)(1)A developer of a nonderivative covered model that is not the subject of a positive safety determination shall submit to the Frontier Model Division an annual certification of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division.
(2)In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A)The nature and magnitude of hazardous capabilities that the covered model possesses or may reasonably possess and the outcome of capability testing required by subdivision (c).
(B)An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered models hazardous capabilities.
(C)Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g)A developer shall report each artificial intelligence safety incident affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division. The notification shall be made in the most expedient time possible and without unreasonable delay and in no event later than 72 hours after learning that an artificial intelligence safety incident has occurred or learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h)(1)Reliance on an unreasonable positive safety determination does not relieve a developer of its obligations under this section.
(2)A positive safety determination is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3)A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A)Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B)By the United States Artificial Intelligence Safety Institute, the Frontier Model Division, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
22604.A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a)Obtain a prospective customers basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1)The identity of that prospective customer.
(2)The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3)The email address and telephonic contact information used to verify a prospective customers identity.
(4)The Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(b)Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
(c)Annually validate the information collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).
(d)Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e)Implement the capability to promptly enact a full shutdown in the event of an emergency.
22605.(a)A developer of a covered model that provides commercial access to that covered model shall provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
(b)A person that operates a computing cluster shall provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developers terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
22606.(a)If the Attorney General has reasonable cause to believe that a person is violating this chapter, the Attorney General shall commence a civil action in a court of competent jurisdiction.
(b)In a civil action under this section, the court may award any of the following:
(1)(A)Preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for a violation of this chapter, including deletion of the covered model and the weights utilized in that model.
(B)Relief pursuant to this paragraph shall be granted only in response to harm or an imminent risk or threat to public safety.
(2)Other relief as the court deems appropriate, including monetary damages to persons aggrieved and an order for the full shutdown of a covered model.
(3)A civil penalty in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation.
(c)In the apportionment of penalties assessed pursuant to this section, defendants shall be jointly and severally liable.
(d)A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section if the court concludes that both of the following are true:
(1)Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
(2)The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
22607.(a)Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a developer shall not prevent an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(b)Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate against an employee for disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603.
(c)The Attorney General may publicly release any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes that doing so will serve the public interest.
(d)Employees shall seek relief for violations of this section pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e)Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a developer shall provide clear notice to all employees working on covered models of their rights and responsibilities under this section.
SEC. 4.Section 11547.6 is added to the Government Code, to read:11547.6.(a)As used in this section:
(1)Hazardous capability has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2)Positive safety determination has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b)The Frontier Model Division is hereby created within the Department of Technology.
(c)The Frontier Model Division shall do all of the following:
(1)Review annual certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2)Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3)(A)Issue guidance, standards, and best practices sufficient to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific requirements on the duties required under Section 22603 of the Business and Professions Code.
(B)Establish an accreditation process and relevant accreditation standards under which third parties may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).
(4)Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5)Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
(6)(A)Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B)The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(7)Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(8)Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A)Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilities.
(B)Advise the Frontier Model Division on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C)Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(9)Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
(10)Monitor relevant developments relating to the safety risks associated with the development of artificial intelligence models and the functioning of markets for artificial intelligence models.
(11)Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(12)(A)Develop and submit to the Judicial Council proposed model jury instructions for actions brought by individuals injured by a hazardous capability of a covered model.
(B)In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i)The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii)Whether and to what extent the developers safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the mandatory safety policies of comparable developers.
(iii)The extent and quality of the developers safety and security protocols prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities.
(iv)Whether and to what extent the developer and its agents complied with the developers safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v)Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state of the art, relevant risks that its model might pose.
(d)There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1)All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2)All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
SEC. 5.Section 11547.7 is added to the Government Code, to read:11547.7.(a)The Department of Technology shall commission consultants, pursuant to subdivision (b), to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes, but is not limited to, all of the following:
(1)A fully owned and hosted cloud platform.
(2)Necessary human expertise to operate and maintain the platform.
(3)Necessary human expertise to support, train, and facilitate use of CalCompute.
(b)The consultants shall include, but not be limited to, representatives of national laboratories, public universities, and any relevant professional associations or private sector stakeholders.
(c)To meet the objective of establishing CalCompute, the Department of Technology shall require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
(1)An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
(2)The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
(3)A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
(4)A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
(5)An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
(6)An analysis of the states investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
(7)A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
(d)The Department of Technology shall submit, pursuant to Section 9795, an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above.
(e)The Department of Technology may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this section.
(f)This section shall become operative only upon an appropriation in a budget act for the purposes of this section.
SEC. 6.The provisions of this act are severable. If any provision of this act or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
SEC. 7.This act shall be liberally construed to effectuate its purposes.
SEC. 8.The duties and obligations imposed by this act are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law.
SEC. 9.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
|
Date
|
Status
|
|---|---|
| Sep 29, 2024 | In Senate. Consideration of Governor's veto pending. |
| Sep 29, 2024 | Vetoed by the Governor. |
| Sep 9, 2024 | Enrolled and presented to the Governor at 3 p.m. |
| Aug 29, 2024 | Assembly amendments concurred in. (Ayes 30. Noes 9. Page 5639.) Ordered to engrossing and enrolling. |
| Aug 28, 2024 | In Senate. Concurrence in Assembly amendments pending. |
| Aug 28, 2024 | Read third time. Passed. (Ayes 48. Noes 16. Page 6719.) Ordered to the Senate. |
| Aug 22, 2024 | Ordered to third reading. |
| Aug 22, 2024 | Read third time and amended. |
| Aug 20, 2024 | Read second time. Ordered to third reading. |
| Aug 19, 2024 | Read second time and amended. Ordered to second reading. |
| Aug 15, 2024 | From committee: Do pass as amended. (Ayes 11. Noes 3.) (August 15). |
| Aug 7, 2024 | August 7 set for first hearing. Placed on suspense file. |
| Jul 3, 2024 | Read second time and amended. Re-referred to Com. on APPR. |
| Jul 2, 2024 | From committee: Do pass as amended and re-refer to Com. on APPR. (Ayes 9. Noes 1.) (July 2). |
| Jun 20, 2024 | Read second time and amended. Re-referred to Com. on JUD. |
| Jun 19, 2024 | From committee: Do pass as amended and re-refer to Com. on JUD. (Ayes 8. Noes 0.) (June 18). |
| Jun 5, 2024 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on P. & C.P. |
| Jun 3, 2024 | Referred to Coms. on P. & C.P. and JUD. |
| May 22, 2024 | In Assembly. Read first time. Held at Desk. |
| May 21, 2024 | Read third time. Passed. (Ayes 32. Noes 1. Page 4091.) Ordered to the Assembly. |
| May 20, 2024 | Read second time. Ordered to third reading. |
| May 16, 2024 | Read second time and amended. Ordered to second reading. |
| May 16, 2024 | From committee: Do pass as amended. (Ayes 5. Noes 0. Page 3969.) (May 16). |
| May 10, 2024 | Set for hearing May 16. |
| May 6, 2024 | May 6 hearing: Placed on APPR suspense file. |
| Apr 30, 2024 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on APPR. |
| Apr 26, 2024 | Set for hearing May 6. |
| Apr 23, 2024 | From committee: Do pass and re-refer to Com. on APPR. (Ayes 11. Noes 0. Page 3721.) (April 23). Re-referred to Com. on APPR. |
| Apr 16, 2024 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on G.O. |
| Apr 16, 2024 | Set for hearing April 23. |
| Apr 8, 2024 | Read second time and amended. Re-referred to Com. on G.O. |
| Apr 4, 2024 | From committee: Do pass as amended and re-refer to Com. on G.O. (Ayes 9. Noes 0. Page 3447.) (April 2). |
| Mar 20, 2024 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on JUD. |
| Mar 5, 2024 | Set for hearing April 2. |
| Feb 14, 2024 | Referred to Coms. on JUD. and G.O. |
| Feb 8, 2024 | From printer. May be acted upon on or after March 9. |
| Feb 7, 2024 | Introduced. Read first time. To Com. on RLS. for assignment. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Aug 29, 2024 | Unfinished Business SB1047 Wiener et al. Concurrence | Senate Floor |
Passed |
|
| Aug 28, 2024 | SB 1047 Wiener Senate Third Reading By Bennett | Assembly Floor |
Passed |
|
| Aug 15, 2024 | Do pass as amended. | Assembly Standing Committee on Appropriations |
Passed |
|
| Jul 2, 2024 | Do pass as amended and be re-referred to the Committee on [Appropriations] | Assembly Standing Committee on Judiciary |
Passed |
|
| Jun 18, 2024 | Do pass as amended and be re-referred to the Committee on [Judiciary] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
|
| May 21, 2024 | Senate 3rd Reading SB1047 Wiener et al. | Senate Floor |
Passed |
|
| May 16, 2024 | Do pass as amended | Senate Standing Committee on Appropriations |
Passed |
|
| May 6, 2024 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Apr 23, 2024 | Do pass, but first be re-referred to the Committee on [Appropriations] | Senate Standing Committee on Governmental Organization |
Passed |
|
| Apr 2, 2024 | Do pass as amended, but first amend, and re-refer to the Committee on [Governmental Organization] | Senate Standing Committee on Judiciary |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| 8vdx | ** Against | |
| Abundance Institute | ** Against | |
| Acclamation Insurance Management Services | ** Against | |
| Adacta Labs | ** Against | |
| AE Studio | ** Support | |
| Aidy | ** Against | |
| AI Safety Student Team | ** Support | |
| Alec Action | ** Against | |
| Algolia | ** Against | |
| Allied Managed Care | ** Against | |
| Alloy Automation | ** Against | |
| American Consumer Institute | ** Against | |
| American Economic Liberties Project | ** Support | |
| Andi | ** Against | |
| Andreessen Horowitz | ** Against | |
| Andy Ai | ** Against | |
| Anneal | ** Against | |
| Answerdotai, INC | ** Against | |
| Antifragile Research Dba Creatorml | ** Against | |
| Apart Research | ** Support | |
| Apriora | ** Against | |
| Argovox | ** Against | |
| Association of National Advertisers | ** Against | |
| Baseline Ai | ** Against | |
| Bilanc Finance INC. | ** Against | |
| Brighterway | ** Against | |
| California Chamber Of Commerce | ** Against | |
| California Fuels And Convenience Alliance | ** Against | |
| California Land Title Association | ** Against | |
| California Life Sciences | ** Against | |
| California Manufacturers And Technology Association | ** Against | |
| California State Council of Service Employees International Union | ** Support | |
| Cambridge Boston Alignment Initiative | ** Support | |
| Candoriq | ** Against | |
| Canonical Ai INC | ** Against | |
| Causative Labs | ** Support | |
| Center For Ai Safety Action Fund | ** Support | |
| Center for Data Innovation | ** Against | |
| Chamber Of Progress | ** Against | |
| Chapman University | ** Support | |
| Chima | ** Against | |
| Circle Medical | ** Against | |
| Citron Labs | ** Against | |
| Civic AI Security Program | ** Support | |
| Civil Justice Association Of California | ** Against | |
| Clarum | ** Against | |
| Clerky | ** Against | |
| Coalition of California Chambers – Orange County | ** Against | |
| Coalition of Small and Disabled Veteran Businesses | ** Against | |
| Cointracker | ** Against | |
| Competitive Enterprise Institute | ** Against | |
| Computer And Communications Industry Association | ** Against | |
| Constructable | ** Against | |
| Consumer Technology Association | ** Against | |
| Consumer Watchdog | ** Support | |
| Context Fund | ** Against | |
| Curtsy | ** Against | |
| Cyble | ** Against | |
| Deconvolute Ai | ** Against | |
| Deferred | ** Against | |
| Denizen | ** Support | |
| Depict.ai | ** Support | |
| Dianahr | ** Against | |
| Digital First Project | ** Against | |
| Distill | ** Against | |
| District Council Of Iron Workers Of The State Of California And Vicinity | ** Support | |
| Docsum | ** Against | |
| Double Finance | ** Against | |
| EcomBack | ** Against | |
| Economic Security Project Action | ** Support | |
| Eden - Workplace Management Software | ** Against | |
| Edge | ** Against | |
| Electronic Frontier Foundation | ** Support | |
| Elicit | ** Support | |
| Ello | ** Against | |
| Elythea | ** Against | |
| Encode Justice | ** Support | |
| Encultured Ai | ** Support | |
| Enh Alpha LLC | ** Support | |
| Entori, INC | ** Against | |
| Envelope | ** Against | |
| Extensional, INC | ** Against | |
| Far AI, INC. | ** Support | |
| Fathom Radiant | ** Support | |
| Fetchflow | ** Against | |
| Figments Corp | ** Against | |
| Firebender Corp | ** Against | |
| Flasher Barricade Association | ** Against | |
| Flexdesk | ** Against | |
| Focal INC | ** Against | |
| Fondo | ** Against | |
| Foresight Institute | ** Support | |
| Forhumanity | ** Support | |
| Future Of Life Institute | ** Support | |
| General Agents | ** Support | |
| General Proximity | ** Support | |
| Gigaml | ** Against | |
| Gladstone AI | ** Support | |
| Gleam | ** Against | |
| ** Against | ||
| Greptile | ** Against | |
| Happyrobot INC. | ** Against | |
| Hazel | ** Against | |
| Hedgy Labs | ** Against | |
| Higher Ground Labs | ** Support | |
| Imbue | ** Support | |
| Indivisible Ca Statestrong | ** Support | |
| Insights Association | ** Against | |
| International Business Machines Corporation | ** Against | |
| International Center for Law & Economics | ** Against | |
| Invert INC | ** Against | |
| Invoke Labs | ** Against | |
| Keywords Ai INC. | ** Against | |
| Kira Center for Ai Risks & Impacts | ** Support | |
| Kontractify, INC | ** Against | |
| K-scale Labs | ** Against | |
| Latino Community Foundation | ** Support | |
| Lightski | ** Against | |
| Linc Ai | ** Against | |
| Linum | ** Against | |
| Lionheart Ventures | ** Support | |
| Listening.com | ** Against | |
| Local Reg | ** Against | |
| Los Angeles Area Chamber Of Commerce | ** Against | |
| Los Angeles Area Chamber Of Commerce | ** Support | |
| Los Angeles Business Federation | ** Against | |
| Los Angeles County Business Federation | ** Against | |
| Loveable Labs Incorporated | ** Support | |
| Lovecast INC Dba Stably Ai | ** Against | |
| Lumona | ** Against | |
| Markprompt | ** Against | |
| Marr Labs Technologies, INC | ** Against | |
| Martian Mobility INC. Dba Telematica | ** Against | |
| Meta | ** Against | |
| Metriport | ** Against | |
| Miru | ** Against | |
| MIT AI Alignment | ** Support | |
| Ml Alignment & Theory Scholars | ** Support | |
| Momentum | ** Support | |
| Movley | ** Against | |
| Mythos Ventures | ** Support | |
| National Federation Of Independent Business | ** Against | |
| New Media Studio | ** Support | |
| Normative | ** Support | |
| Oakland Privacy | ** Support | |
| Octolane Ai | ** Against | |
| Oneshop | ** Against | |
| Onward | ** Against | |
| Openmart | ** Against | |
| Openpod | ** Against | |
| Orange Collective | ** Against | |
| Orange County Business Council | ** Against | |
| Pabio INC. | ** Against | |
| Panoplia Laboratories | ** Support | |
| Paper Farms | ** Support | |
| Patched Codes, INC | ** Against | |
| Patika Technology, INC | ** Against | |
| Patterns | ** Against | |
| Persist Ai | ** Against | |
| Phonely | ** Against | |
| Picnichealth | ** Against | |
| Playground | ** Against | |
| Poll Everywhere | ** Against | |
| Preloop INC. | ** Against | |
| Q8 Empowering Change | ** Against | |
| Query Vary | ** Against | |
| Rainforest Qa, INC. | ** Against | |
| Redwood Research | ** Support | |
| Regology INC | ** Against | |
| Rejuvenation Technologies INC | ** Against | |
| Retell Ai | ** Against | |
| Revamp Ai | ** Against | |
| Reworkd Ai | ** Against | |
| Rinsed | ** Against | |
| Rippling | ** Against | |
| Risotto | ** Against | |
| Rocketlit INC. | ** Against | |
| Rootly | ** Against | |
| R Street Institute | ** Against | |
| Safe AI Future | ** Support | |
| San Diego Regional Chamber Of Commerce | ** Against | |
| San Juan Capistrano Chamber of Commerce | ** Against | |
| Sela | ** Against | |
| Sepal Ai | ** Against | |
| Sideguide Technologies INC | ** Against | |
| Silicon Valley Leadership Group | ** Against | |
| Simplify | ** Against | |
| Software And Information Industry Association | ** Against | |
| Sorting Robotics | ** Against | |
| Southwest California Legislative Council | ** Against | |
| Spoken | ** Against | |
| Sprx | ** Against | |
| Squire Ai | ** Against | |
| Stack Ai | ** Against | |
| Sterling Road | ** Against | |
| Storyquest, INC. | ** Against | |
| Storyworth | ** Against | |
| Sway Finance | ** Against | |
| Sync Labs | ** Against | |
| Synova Life Sciences | ** Against | |
| Taxgpt INC | ** Against | |
| TechEquity Collaborative | ** Support | |
| Technet | ** Against | |
| Tensor | ** Against | |
| Tesorio INC | ** Against | |
| The Future Society | ** Support | |
| Tracecat | ** Against | |
| Trainy | ** Against | |
| Truewind Technologies, INC. | ** Against | |
| Turbolayer INC. Dba Automatica | ** Against | |
| Turboprop | ** Against | |
| Turing Labs | ** Against | |
| Tusk | ** Against | |
| Twine | ** Against | |
| Two Dots | ** Against | |
| Up Dog | ** Against | |
| Upkeep | ** Against | |
| Viva Labs | ** Against | |
| Walrus Tech INC | ** Against | |
| White Space Marketing Group | ** Support | |
| Willow | ** Against | |
| Wordware | ** Against | |
| Y Combinator | ** Against | |
| Zapier | ** Against | |
| Zaymo | ** Against | |
| Zep Ai | ** Against |
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 1047 |
VETO
Bill No: SB 1047
Author: Wiener (D), et al.
Enrolled: 9/3/24
Vote: 27
SENATE JUDICIARY COMMITTEE: 9-0, 4/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Min, Stern, Wahab
NO VOTE RECORDED: Wilk, Niello
SENATE GOVERNMENTAL ORG. COMMITTEE: 11-0, 4/23/24
AYES: Dodd, Wilk, Ashby, Bradford, Glazer, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Smallwood-Cuevas
NO VOTE RECORDED: Alvarado-Gil, Archuleta, Jones, Nguyen, Seyarto
SENATE APPROPRIATIONS COMMITTEE: 5-0, 5/16/24
AYES: Caballero, Ashby, Becker, Bradford, Wahab
NO VOTE RECORDED: Jones, Seyarto
SENATE FLOOR: 32-1, 5/21/24
AYES: Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
NOES: Grove
NO VOTE RECORDED: Allen, Alvarado-Gil, Dahle, Jones, Nguyen, Niello, Seyarto
SENATE FLOOR: 30-9, 8/29/24
AYES: Allen, Archuleta, Ashby, Atkins, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Newman, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
NOES: Alvarado-Gil, Becker, Dahle, Grove, Jones, Nguyen, Niello, Ochoa Bogh, Seyarto
NO VOTE RECORDED: Min
ASSEMBLY FLOOR: 48-16, 8/28/24 - See last page for vote
SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
SOURCE: Center for AI Safety Action Fund
Economic Security Project Action
Encode Justice
DIGEST:
This bill requires developers of powerful artificial intelligence models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. This bill establishes a state entity to oversee the development of these models and calls for the creation of a consortium to develop a framework for a public cloud computing cluster.
ANALYSIS:
Existing law:
- Establishes the California Department of Technology (CDT) within the Government Operations Agency (GovOps), under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Government Code Section 11545(a).)
- Provides that persons are responsible, not only for the result of their willful acts, but also for an injury occasioned to another by their want of ordinary care or skill in the management of their property or person, except so far as the latter has, willfully or by want of ordinary care, brought the injury upon themselves. (Civil Code § 1714(a).)
This bill:
- Establishes the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
- Requires developers of "covered models"—AI models trained or fine-tuned with a specified level of compute power and that meet specified cost thresholds—and certain derivative models to, among other things:
- Before training the model, implement certain cybersecurity protections, the capability to promptly enact a full shutdown, a written safety and security protocol, and take reasonable care to implement appropriate measures to prevent "critical harms"—defined as mass casualties, at least $500 million in damage, or other comparable harms.
- Before using the model or making it publicly available: assess whether the model is reasonably capable of causing or materially enabling a critical harm; record and retain test results from the assessment; and take reasonable care to implement appropriate safeguards.
- Neither use, unless exclusively for training or evaluation, nor make publicly available a model if there is an unreasonable risk it will cause or materially enable a critical harm.
- Beginning in 2026, annually retain a third-party auditor to perform an independent audit of a developer's compliance with applicable duties.
- Make public and provide to the Attorney General redacted copies of safety and security protocol and auditors' reports. Upon request, provide to the Attorney General unredacted copies of those documents, which are exempt from the California Public Records Act. Submit annual compliance statements to the Attorney General. Report to the Attorney General safety incidents within 72 hours.
- Requires a person operating a computing cluster to implement appropriate written policies and procedures to take specified actions when a customer utilizes compute resources that would be sufficient to train a covered model, including obtaining a prospective customer’s identifying information and information to assess whether the customer intends to use the cluster to train a covered model.
- Authorizes the Attorney General to bring a civil action for violations of the bill for specified civil penalties and injunctive and declaratory relief.
- Provides whistleblower protections for employees including by prohibiting a developer of a covered model from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
- Creates the Board of Frontier Models within GovOps, consisting of nine members, five appointed by the Governor and four by the Legislature and establishes certain qualifications and requirements for members of the board.
- Requires GovOps, by 2027, to issue guidance for preventing unreasonable risks and adopt regulations to update the definition of a "covered entity" and establish auditing standards, subject to approval by the Board of Frontier Models.
- Establishes, upon appropriation, in GovOps, a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as "CalCompute" that advances the development and deployment of AI, as prescribed.
- Provides definitions for the relevant terms.
Background
Owing to recent advances in processing power and the rise of big data, artificial intelligence’s (AI) capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact business, and consume media. It has been used to accelerate productivity, achieve efficiencies, liberate us from drudgery, write our college essay, help us understand and enjoy the world, upgrade the Pope’s fashion, connect with each other, and live longer, fuller lives. It has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy. For good or ill, its transformative potential seems boundless.
This bill seeks to establish guardrails for the development of the most powerful AI models to avoid the more catastrophic possibilities about which experts have raised alarms. It places a series of obligations on developers of “covered models” and providers of the cloud compute for training such models. This bill also seeks to establish a framework for developing a public cloud-computing cluster that facilitates equitable participation in the development and deployment of responsible AI systems. This bill is co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. It is supported by a host of tech companies and labor organizations, as well as the Los Angeles Area Chamber of Commerce, and opposed by the Chamber of Progress and a coalition of industry associations.
Comments
According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- CDT indicates significant one-time costs and ongoing costs (General Fund) to implement this bill.
- Judicial Council estimates minor and absorbable costs.
- Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to the courts.
- Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill.
According to the Assembly Appropriations Committee:
- Costs (General Fund) to GovOps to establish and operate the FMD and the Board. GovOps reports ongoing annual costs between $5 million and $10 million.
- Costs (General Fund) to CDT to establish and operate CalCompute. In the first year of implementation, CDT reports costs of $508,000 for two temporary positions, $6 million for external consultants, and $1 million for GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $3.7 million for 16 additional positions, $3 million for external consultants, and $700,000 for GenAI talent practices. The bill authorizes CDT to receive private donations, grants, and local funds to support CalCompute, which may offset General Fund costs by an unknown amount.
CDT notes it is challenging to hire GenAI talent in the public sector since salaries are often two or three times higher in the private sector, and private sector employers can offer bonuses, equity, and stock options. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees. GovOps will likely also face these challenges and additional cost pressures.
- Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Department of Justice (DOJ) and Labor Commissioner to enforce violations of the bill. Costs to DOJ may be in the high hundreds of thousands to low millions of dollars annually. Actual costs will depend on the number of violations, the number of actions filed, and the amount of workload associated with each action.
- Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate violations of the bill and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The fiscal year 2024-25 budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
- Minor and absorbable costs to Judicial Council to review and process the model jury instructions recommended by the FMD, as required by the bill.
According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years.
SUPPORT: (Verified 10/7/24)
Center for AI Safety Action Fund (co-source)
Economic Security Project Action (co-source)
Encode Justice (co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program
Denizen
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Elicit
Enh Alpha LLC
Far AI, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group
OPPOSITION: (Verified 10/7/24)
8vdx
Abundance Institute
Adacta Labs
Aidy
Alec Action
Algolia
Alloy Automation
American Consumer Institute
Andi
Andy AI
Anneal
Antifragile Research Dba Creatorml
Apriora
Argovox
Association of National Advertisers
Baseline AI
Bilanc Finance INC.
Brighterway
California Chamber of Commerce
California Manufacturers and Technology Association
Candoriq
Canonical AI INC
Center for Data Innovation
Chamber of Progress
Chima
Circle Medical
Citron Labs
Civil Justice Association of California
Clarum
Clerky
Cointracker
Competitive Enterprise Institute
Computer and Communications Industry Association
Constructable
Consumer Technology Association
Context Fund
Curtsy
Cyble
Deconvolute AI
Deferred
Dianahr
Digital First Project
Distill
Docsum
Double
Double Finance
Eden - Workplace Management Software
Edge
Ello
Elythea
Entori, INC
Envelope
Extensional, INC
Fetchflow
Figments Corp
Firebender Corp
Flexdesk
Focal INC
Fondo
Gigaml
Gleam
Greptile
Happyrobot INC.
Hazel
Hedgy Labs
Insights Association
International Center for Law & Economics
Invert INC
Invoke Labs
Jo
K-scale Labs
Keywords Ai INC.
Kontractify, INC
Lightski
Linc AI
Linum
Listening.com
Local Reg
Los Angeles Area Chamber of Commerce
Lovecast INC Dba Stably AI
Lumona
Markprompt
Marr Labs Technologies, INC
Martian Mobility INC. Dba Telematica
Mayor of City & County of San Francisco London Breed
Metriport
Miru
Movley
Octolane AI
Oneshop
Onward
Openmart
Openpod
Orange Collective
Pabio INC.
Patched Codes, INC
Patika Technology, INC
Patterns
Persist AI
Phonely
Picnichealth
Playground
Poll Everywhere
Preloop INC.
Q8 Empowering Change
Query Vary
R Street Institute
Rainforest Qa, INC.
Regology INC
Rejuvenation Technologies INC
Retell AI
Revamp AI
Reworkd AI
Rinsed
Rippling
Risotto
Rocketlit INC.
Rootly
Sela
Sepal AI
Sideguide Technologies INC (known As Firecrawl)
Silicon Valley Leadership Group
Simplify
Software and Information Industry Association
Sorting Robotics
Spoken
Sprx
Squire AI
Stack AI
Sterling Road
Storyquest, INC.
Storyworth
Sway Finance
Sync Labs
Synova Life Sciences
Tailor
Taxgpt INC
Technet
Tensor
Tesorio INC
Tracecat
Trainy
Truewind Technologies, INC.
Turbolayer INC. Dba Automatica
Turboprop
Turing Labs
Tusk
Twine
Two Dots
Up Dog
Upkeep
Viva Labs
Walrus Tech INC
Willow
Wordware
Y Combinator
Zapier
Zaymo
Zep AI
ARGUMENTS IN SUPPORT: Encode Justice, a co-sponsor of the bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as “frontier AI models.” These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue.
These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden’s Executive Order on Artificial Intelligence for their significant national security and public safety implications.
ARGUMENTS IN OPPOSITION: Chamber of Progress writes:
Unfortunately, SB 1047 forces model developers to engage in speculative fiction about imagined threats of machines run amok, computer models spun out of control, and other nightmare scenarios for which there is no basis in reality.
Instead, SB 1047 forces developers operating in the real world to proactively mitigate against every conceivable harm - and many inconceivable ones - not just by the model itself, but subsequent third parties who make use of the model.
GOVERNOR'S VETO MESSAGE:
This bill would require developers of large artificial intelligence (AI) models, and those providing the computing power to train such models, to put certain safeguards and policies in place to prevent catastrophic harm. The bill would also establish the Board of Frontier Models - a state entity - to oversee the development of these models.
California is home to 32 of the world's 50 leading AI companies, pioneers in one of the most significant technological advances in modern history. We lead in this space because of our research and education institutions, our diverse and motivated workforce, and our free-spirited cultivation of intellectual freedom. As stewards and innovators of the future, I take seriously the responsibility to regulate this industry.
This year, the Legislature sent me several thoughtful proposals to regulate AI companies in response to current, rapidly evolving risks - including threats to our democratic process, the spread of misinformation and deepfakes, risks to online privacy, threats to critical infrastructure, and disruptions in the workforce. These bills, and actions by my Administration, are guided by principles of accountability, fairness, and transparency of AI systems and deployment of AI technology in California.
SB 1047 magnified the conversation about threats that could emerge from the deployment of AI. Key to the debate is whether the threshold for regulation should be based on the cost and number of computations needed to develop an AI model, or whether we should evaluate the system's actual risks regardless of these factors. This global discussion is occurring as the capabilities of AI continue to scale at an impressive pace. At the same time, the strategies and solutions for addressing the risk of catastrophic harm are rapidly evolving.
By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.
Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.
Let me be clear - I agree with the author - we cannot afford to wait for a major catastrophe to occur before taking action to protect the public. California will not abandon its responsibility. Safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable. I do not agree, however, that to keep the public safe, we must settle for a solution that is not informed by an empirical trajectory analysis of AI systems and capabilities. Ultimately, any framework for effectively regulating AI needs to keep pace with the technology itself.
To those who say there's no problem here to solve, or that California does not have a role in regulating potential national security implications of this technology, I disagree. A California-only approach may well be warranted - especially absent federal action by Congress - but it must be based on empirical evidence and science. The U.S. AI Safety Institute, under the National Institute of Science and Technology, is developing guidance on national security risks, informed by evidence-based approaches, to guard against demonstrable risks to public safety. Under an Executive Order I issued in September 2023, agencies within my Administration are performing risk analyses of the potential threats and vulnerabilities to California's critical infrastructure using AI. These are just a few examples of the many endeavors underway, led by experts, to inform policymakers on AI risk management practices that are rooted in science and fact. And endeavors like these have led to the introduction of over a dozen bills regulating specific, known risks posed by AI, that I have signed in the last 30 days.
I am committed to working with the Legislature, federal partners, technology experts, ethicists, and academia, to find the appropriate path forward, including legislation and regulation. Given the stakes - protecting against actual threats without unnecessarily thwarting the promise of this technology to advance the public good - we must get this right.
For these reasons, I cannot sign this bill.
ASSEMBLY FLOOR: 48-16, 8/28/24
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman, Bonta, Bryan, Calderon, Juan Carrillo, Connolly, Mike Fong, Friedman, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Holden, Jones-Sawyer, Kalra, Lee, Lowenthal, Maienschein, Mathis, McKinnor, Muratsuchi, Pacheco, Papan, Pellerin, Ramos, Rendon, Reyes, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Waldron, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas
NOES: Alanis, Bains, Davies, Dixon, Hoover, Irwin, Low, McCarty, Jim Patterson, Joe Patterson, Petrie-Norris, Luz Rivas, Sanchez, Ta, Valencia, Wallis
NO VOTE RECORDED: Boerner, Wendy Carrillo, Cervantes, Chen, Megan Dahle, Essayli, Flora, Gallagher, Jackson, Lackey, Stephanie Nguyen, Ortega, Quirk-Silva, Rodriguez, Villapudua
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
10/10/24 10:24:34
**** END ****
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 1047 |
UNFINISHED BUSINESS
Bill No: SB 1047
Author: Wiener (D), et al.
Amended: 8/22/24
Vote: 21
SENATE JUDICIARY COMMITTEE: 9-0, 4/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Min, Stern, Wahab
NO VOTE RECORDED: Wilk, Niello
SENATE GOVERNMENTAL ORG. COMMITTEE: 11-0, 4/23/24
AYES: Dodd, Wilk, Ashby, Bradford, Glazer, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Smallwood-Cuevas
NO VOTE RECORDED: Alvarado-Gil, Archuleta, Jones, Nguyen, Seyarto
SENATE APPROPRIATIONS COMMITTEE: 5-0, 5/16/24
AYES: Caballero, Ashby, Becker, Bradford, Wahab
NO VOTE RECORDED: Jones, Seyarto
SENATE FLOOR: 32-1, 5/21/24
AYES: Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
NOES: Grove
NO VOTE RECORDED: Allen, Alvarado-Gil, Dahle, Jones, Nguyen, Niello, Seyarto
ASSEMBLY FLOOR: 48-16, 8/28/24 - See last page for vote
SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
SOURCE: Center for AI Safety Action Fund
Economic Security Project Action
Encode Justice
DIGEST:
This bill requires developers of powerful artificial intelligence models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. This bill establishes a state entity to oversee the development of these models and calls for the creation of a consortium to develop a framework for a public cloud computing cluster.
Assembly Amendments
restructure the bill and drastically narrow the requirements imposed on developers.
ANALYSIS:
Existing law:
- Establishes the California Department of Technology (CDT) within the Government Operations Agency (GovOps), under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Gov. Code Sec. 11545(a).)
- Provides that persons are responsible, not only for the result of their willful acts, but also for an injury occasioned to another by their want of ordinary care or skill in the management of their property or person, except so far as the latter has, willfully or by want of ordinary care, brought the injury upon themselves. (Civ. Code § 1714(a).)
This bill:
- Establishes the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
- Requires developers of "covered models"—AI models trained or fine-tuned with a specified level of compute power and that meet specified cost thresholds—and certain derivative models to, among other things:
- Before training the model, implement certain cybersecurity protections, the capability to promptly enact a full shutdown, a written safety and security protocol, and take reasonable care to implement appropriate measures to prevent "critical harms"—defined as mass casualties, at least $500 million in damage, or other comparable harms.
- Before using the model or making it publicly available: assess whether the model is reasonably capable of causing or materially enabling a critical harm; record and retain test results from the assessment; and take reasonable care to implement appropriate safeguards.
- Neither use, unless exclusively for training or evaluation, nor make publicly available a model if there is an unreasonable risk it will cause or materially enable a critical harm.
- Beginning in 2026, annually retain a third-party auditor to perform an independent audit of a developer's compliance with applicable duties.
- Make public and provide to the Attorney General redacted copies of safety and security protocol and auditors' reports. Upon request, provide to the Attorney General unredacted copies of those documents, which are exempt from the California Public Records Act. Submit annual compliance statements to the Attorney General. Report to the Attorney General safety incidents within 72 hours.
- Requires a person operating a computing cluster to implement appropriate written policies and procedures to take specified actions when a customer utilizes compute resources that would be sufficient to train a covered model, including obtaining a prospective customer’s identifying information and information to assess whether the customer intends to use the cluster to train a covered model.
- Authorizes the Attorney General to bring a civil action for violations of the bill for specified civil penalties and injunctive and declaratory relief.
- Provides whistleblower protections for employees including by prohibiting a developer of a covered model from preventing an employee from disclosing information, or retaliating against an employee for disclosing information, to the Attorney General or Labor Commissioner if the employee has reasonable cause to believe the information indicates the developer is out of compliance with certain requirements or that the covered model poses an unreasonable risk of critical harm.
- Creates the Board of Frontier Models within GovOps, consisting of nine members, five appointed by the Governor and four by the Legislature and establishes certain qualifications and requirements for members of the board.
- Requires GovOps, by 2027, to issue guidance for preventing unreasonable risks and adopt regulations to update the definition of a "covered entity" and establish auditing standards, subject to approval by the Board of Frontier Models.
- Establishes, upon appropriation, in GovOps, a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as "CalCompute" that advances the development and deployment of AI, as prescribed.
- Provides definitions for the relevant terms.
Background
Owing to recent advances in processing power and the rise of big data, artificial intelligence’s (AI) capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact business, and consume media. It has been used to accelerate productivity, achieve efficiencies, liberate us from drudgery, write our college essay, help us understand and enjoy the world, upgrade the Pope’s fashion, connect with each other, and live longer, fuller lives. It has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy. For good or ill, its transformative potential seems boundless.
This bill seeks to establish guardrails for the development of the most powerful AI models to avoid the more catastrophic possibilities about which experts have raised alarms. It places a series of obligations on developers of “covered models” and providers of the cloud compute for training such models. This bill also seeks to establish a framework for developing a public cloud-computing cluster that facilitates equitable participation in the development and deployment of responsible AI systems. This bill is co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. It is supported by a host of tech companies and labor organizations, as well as the Los Angeles Area Chamber of Commerce, and opposed by the Chamber of Progress and a coalition of industry associations.
Comments
According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- CDT indicates significant one-time costs and ongoing costs (General Fund) to implement this bill.
- Judicial Council estimates minor and absorbable costs.
- Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to the courts.
- Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill.
According to the Assembly Appropriations Committee:
- Costs (General Fund) to GovOps to establish and operate the FMD and the Board. GovOps reports ongoing annual costs between $5 million and $10 million.
- Costs (General Fund) to CDT to establish and operate CalCompute. In the first year of implementation, CDT reports costs of $508,000 for two temporary positions, $6 million for external consultants, and $1 million for GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $3.7 million for 16 additional positions, $3 million for external consultants, and $700,000 for GenAI talent practices. The bill authorizes CDT to receive private donations, grants, and local funds to support CalCompute, which may offset General Fund costs by an unknown amount.
CDT notes it is challenging to hire GenAI talent in the public sector since salaries are often two or three times higher in the private sector, and private sector employers can offer bonuses, equity, and stock options. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees. GovOps will likely also face these challenges and additional cost pressures.
- Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Department of Justice (DOJ) and Labor Commissioner to enforce violations of the bill. Costs to DOJ may be in the high hundreds of thousands to low millions of dollars annually. Actual costs will depend on the number of violations, the number of actions filed, and the amount of workload associated with each action.
- Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate violations of the bill and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The fiscal year 2024-25 budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
- Minor and absorbable costs to Judicial Council to review and process the model jury instructions recommended by the FMD, as required by the bill.
According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years.
SUPPORT: (Verified 8/28/24)
Center for AI Safety Action Fund (co-source)
Economic Security Project Action (co-source)
Encode Justice (co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program
Denizen
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Elicit
Enh Alpha LLC
Far AI, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group
OPPOSITION: (Verified 8/28/24)
8vdx
Abundance Institute
Adacta Labs
Aidy
Alec Action
Algolia
Alloy Automation
American Consumer Institute
Andi
Andy AI
Anneal
Antifragile Research Dba Creatorml
Apriora
Argovox
Association of National Advertisers
Baseline AI
Bilanc Finance INC.
Brighterway
California Chamber of Commerce
California Manufacturers and Technology Association
Candoriq
Canonical AI INC
Center for Data Innovation
Chamber of Progress
Chima
Circle Medical
Citron Labs
Civil Justice Association of California
Clarum
Clerky
Cointracker
Competitive Enterprise Institute
Computer and Communications Industry Association
Constructable
Consumer Technology Association
Context Fund
Curtsy
Cyble
Deconvolute AI
Deferred
Dianahr
Digital First Project
Distill
Docsum
Double
Double Finance
Eden - Workplace Management Software
Edge
Ello
Elythea
Entori, INC
Envelope
Extensional, INC
Fetchflow
Figments Corp
Firebender Corp
Flexdesk
Focal INC
Fondo
Gigaml
Gleam
Greptile
Happyrobot INC.
Hazel
Hedgy Labs
Insights Association
International Center for Law & Economics
Invert INC
Invoke Labs
Jo
K-scale Labs
Keywords Ai INC.
Kontractify, INC
Lightski
Linc AI
Linum
Listening.com
Local Reg
Los Angeles Area Chamber of Commerce
Lovecast INC Dba Stably AI
Lumona
Markprompt
Marr Labs Technologies, INC
Martian Mobility INC. Dba Telematica
Mayor of City & County of San Francisco London Breed
Metriport
Miru
Movley
Octolane AI
Oneshop
Onward
Openmart
Openpod
Orange Collective
Pabio INC.
Patched Codes, INC
Patika Technology, INC
Patterns
Persist AI
Phonely
Picnichealth
Playground
Poll Everywhere
Preloop INC.
Q8 Empowering Change
Query Vary
R Street Institute
Rainforest Qa, INC.
Regology INC
Rejuvenation Technologies INC
Retell AI
Revamp AI
Reworkd AI
Rinsed
Rippling
Risotto
Rocketlit INC.
Rootly
Sela
Sepal AI
Sideguide Technologies INC (known As Firecrawl)
Silicon Valley Leadership Group
Simplify
Software and Information Industry Association
Sorting Robotics
Spoken
Sprx
Squire AI
Stack AI
Sterling Road
Storyquest, INC.
Storyworth
Sway Finance
Sync Labs
Synova Life Sciences
Tailor
Taxgpt INC
Technet
Tensor
Tesorio INC
Tracecat
Trainy
Truewind Technologies, INC.
Turbolayer INC. Dba Automatica
Turboprop
Turing Labs
Tusk
Twine
Two Dots
Up Dog
Upkeep
Viva Labs
Walrus Tech INC
Willow
Wordware
Y Combinator
Zapier
Zaymo
Zep AI
ARGUMENTS IN SUPPORT: Encode Justice, a co-sponsor of the bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as “frontier AI models.” These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue.
These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden’s Executive Order on Artificial Intelligence for their significant national security and public safety implications.
ARGUMENTS IN OPPOSITION: Chamber of Progress writes:
Unfortunately, SB 1047 forces model developers to engage in speculative fiction about imagined threats of machines run amok, computer models spun out of control, and other nightmare scenarios for which there is no basis in reality.
Instead, SB 1047 forces developers operating in the real world to proactively mitigate against every conceivable harm - and many inconceivable ones - not just by the model itself, but subsequent third parties who make use of the model.
ASSEMBLY FLOOR: 48-16, 8/28/24
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman, Bonta, Bryan, Calderon, Juan Carrillo, Connolly, Mike Fong, Friedman, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Holden, Jones-Sawyer, Kalra, Lee, Lowenthal, Maienschein, Mathis, McKinnor, Muratsuchi, Pacheco, Papan, Pellerin, Ramos, Rendon, Reyes, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Waldron, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas
NOES: Alanis, Bains, Davies, Dixon, Hoover, Irwin, Low, McCarty, Jim Patterson, Joe Patterson, Petrie-Norris, Luz Rivas, Sanchez, Ta, Valencia, Wallis
NO VOTE RECORDED: Boerner, Wendy Carrillo, Cervantes, Chen, Megan Dahle, Essayli, Flora, Gallagher, Jackson, Lackey, Stephanie Nguyen, Ortega, Quirk-Silva, Rodriguez, Villapudua
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
8/28/24 22:40:24
**** END ****
SENATE THIRD READING
STR Bill Id:SB 1047
Author:(Wiener)
As Amended Ver:August 22, 2024
Majority vote
SUMMARY
Requires developers of the most advanced, costly artificial intelligence (AI) systems to implement certain protocols and safeguards in order to mitigate the risk of catastrophic harms. Requires operators of computing clusters to obtain specified information relating to customers. Provides for whistleblower protections and enforcement by the Attorney General (AG). Creates the Board of Frontier Models (board), which approves specified regulations and guidance to be issued by Government Operations Agency. Provides, upon appropriation, for the creation of a framework to create a public cloud computing cluster.
Major Provisions
- Requires developers of "covered models"—AI models trained or fine-tuned with a specified level of compute power and that meet specified cost thresholds—and certain derivative models to, among other things:
- Before training the model, implement certain cybersecurity protections, the capability to promptly enact a full shutdown, a written safety and security protocol, and take reasonable care to implement appropriate measures to prevent "critical harms"—defined as mass casualties, at least $500 million in damage, or other comparable harms.
- Before using the model or making it publicly available: assess whether the model is reasonably capable of causing or materially enabling a critical harm; record and retain test results from the assessment; and take reasonable care to implement appropriate safeguards.
- Neither use, unless exclusively for training or evaluation, nor make publicly available a model if there is an unreasonable risk it will cause or materially enable a critical harm.
- Beginning in 2026, annually retain a third-party auditor to perform an independent audit of a developer's compliance with applicable duties.
- Make public and provide to the AG redacted copies of safety and security protocol and auditors' reports. Upon request, provide to the AG unredacted copies of those documents, which are exempt from the California Public Records Act. Submit annual compliance statements to the AG. Report to the AG safety incidents within 72 hours.
- Requires a person operating a computing cluster to implement written policies and procedures to take certain actions when a customer uses compute resources sufficient to train a covered model, including obtaining information to identify the customer.
- Authorizes the AG to bring a civil action for violations of the bill for specified civil penalties and injunctive and declaratory relief.
- Provides whistleblower protections for employees who have reasonable cause to disclose to the AG or Labor Commissioner information regarding a developer's noncompliance with the bill or regarding unreasonably dangerous models.
- Creates the board within the Government Operations Agency, consisting of nine members, five appointed by the Governor and four by the Legislature. Establishes certain qualifications and requirements for members of the board.
- Requires the Government Operations Agency, by 2027, to issue guidance for preventing unreasonable risks and adopt regulations to update the definition of a "covered entity" and establish auditing standards, subject to approval by the board.
- Upon appropriation, establishes in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as "CalCompute" that advances the development and deployment of AI, as prescribed.
COMMENTS
With rapid advances in AI, industry insiders—often bound by punitive non-disclosure agreements—have sounded the alarm regarding the potential risks posed by these technologies. A group of current and former employees at frontier AI companies recently wrote: "These risks range from the further entrenchment of existing inequalities, to manipulation and misinformation, to the loss of control of autonomous AI systems potentially resulting in human extinction." (Emphasis added.)
This bill seeks to mitigate the risk of catastrophic harms from AI models so advanced that they are not yet known to exist. The bill would require developers of such models to create and implement safety and security protocols before initiating training. Developers must also implement the capability to promptly shutdown models. Following training, developers would be required to perform risk assessments on their models and implement reasonable safeguards, subject to third-party auditing, before using or releasing them. If there is an unreasonable risk a model will cause or materially enable a critical harm—mass casualties, at least $500 million in damage, or other comparable harms—developers are prohibited from releasing or using the model.
The bill requires the Government Operations Agency to adopt certain regulations and guidance, subject to approval by the Board of Frontier Models, which would be established under the bill. The bill also adds whistleblower protections; requires operators of computer clusters to implement "know your customer" requirements, including the ability to shut down any resources being used to train an advanced AI model; and lays the groundwork for the creation of a public computing cluster known as "CalCompute." The Attorney General is charged with civil enforcement of the bill's requirements.
This bill has generated a great deal of commentary, consternation, and misconception. To set the record straight: SB 1047's requirements are not onerous. For one, they only impact models that cost over $100 million in computing power to train—this threshold is baked into the definition of "covered model" provided by the bill and cannot be altered except by future legislative action. Secondly, the safety and security protocols developers would be required to implement broadly align with national and international guidelines, as well as procedures these developers already claim to implement. Finally, the bill only requires that developers implement shutdown capabilities for models they themselves control. The open-source community can rest easy knowing the models they download will not contain an immutable kill switch.
This bill does not create a state-sponsored licensing regime for AI; it does not ban the creation and use of AI above a certain compute threshold; it does not impose strict liability on developers for downstream harms caused by third parties, nor alter baseline rules of tort and criminal liability; and it does not create exorbitant costs for startups seeking to train large models. Performing basic risk assessments and implementing reasonable safeguards before training, using, or releasing powerful, generally-capable models—and prohibiting their use only in extreme cases involving unreasonable risks of mass casualties or massive economic damages—is the bare minimum that Californians should expect of an industry claiming to have their best interests at heart.
According to the Author
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we've seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians' lives, without compromising safety or security.
SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a public AI research cluster that will allow startups, researchers, and community groups to participate in the development of large-scale AI systems. By providing a broad range of stakeholders with access to the AI development process, CalCompute will help align large-scale AI systems with the values and needs of California communities.
Arguments in Support
The Center for AI Safety Action Fund, a co-sponsor of this bill, writes:
If California does not act to establish a clear and sensible governance framework, the safety of its citizens could be imperiled, the nation's security could be seriously harmed, and AI's enormous potential to improve our world could be derailed. Placing sensible guardrails around serious risks that the most powerful systems might pose, while taking significant steps towards leveling the playing field for academics and startups, is the best way to ensure that CA's citizens can realize the immense benefits of this technology.
Encode Justice, a co-sponsor of this bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as "frontier AI models." These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue. These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden's Executive Order on Artificial Intelligence for their significant national security and public safety implications.
Arguments in Opposition
California Chamber of Commerce writes on behalf of a coalition of trade associations:
In addition to creating inconsistencies with federal regulations, the bill demands compliance with various vague and impractical, if not technically infeasible, requirements for which developers will be subject to harsh penalties, including potential criminal liability. We are concerned that the bill regulates AI technology as opposed to its high-risk applications, creates significant regulatory uncertainty and therefore high compliance costs, and poses significant liability risks to developers for failing to foresee and block any harmful use of their models by others – all of which inevitably discourages economic and technological innovation. And while recent amendments take important steps in responding to the open-source community, we remain concerned about the impact of the bill on AI research and development in California and the impact on startups. Overall, the bill still makes AI business too risky in California, particularly given the potential penalties under SB 1047.
FISCAL COMMENTS
- Costs (General Fund) to GovOps of an unknown but potentially significant amount to establish and operate the Board, and to issue and update regulations. The bill also requires GovOps to, upon appropriation by the Legislature, establish a consortium and develop a framework for the CalCompute public cloud computing cluster. Although these responsibilities are contingent upon appropriation, they add additional, potentially significant General Fund cost pressure to GovOps.
- Costs (General Fund) to DOJ of an unknown but potentially significant amount to maintain and review developer statements of compliance, and to bring civil enforcement actions. Actual costs will depend in part on the number of violations, the number of actions filed, and the amount of workload associated with each action.
- Costs (Labor and Enforcement Compliance Fund) to the Labor Commissioner of an unknown but amount to receive and act on whistleblower reports.
- Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate violations of the bill and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The fiscal year 2024-25 budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
VOTES
SENATE FLOOR: 32-1-7
YES: Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
NO: Grove
ABS, ABST OR NV: Allen, Alvarado-Gil, Dahle, Jones, Nguyen, Niello, Seyarto
ASM PRIVACY AND CONSUMER PROTECTION: 8-0-3
YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson
ABS, ABST OR NV: Joe Patterson, Dixon, Hoover
ASM JUDICIARY: 9-1-2
YES: Kalra, Bauer-Kahan, Bryan, Connolly, Haney, Maienschein, McKinnor, Pacheco, Reyes
NO: Dixon
ABS, ABST OR NV: Joe Patterson, Sanchez
ASM APPROPRIATIONS: 11-3-1
YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart, Pellerin, Villapudua
NO: Sanchez, Dixon, Ta
ABS, ABST OR NV: Jim Patterson
UPDATED
VERSION: August 22, 2024
CONSULTANT: Josh Tosney / Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0004462
SENATE THIRD READING
STR Bill Id:SB 1047
Author:(Wiener)
As Amended Ver:August 19, 2024
Majority vote
SUMMARY
Requires developers of the most advanced, costly artificial intelligence (AI) systems to implement certain protocols and safeguards in order to mitigate the risk of catastrophic harms. Requires operators of computing clusters to obtain specified information relating to customers. Provides for whistleblower protections and enforcement by the Attorney General (AG). Creates the Board of Frontier Models, which approves specified regulations and guidance pertaining to be issued by Government Operations Agency. Provides, upon appropriation, for the creation of a framework to create a public cloud computing cluster.
Major Provisions
- Requires developers of "covered models"—AI models trained or fine-tuned with a specified level of compute power and that meet specified cost thresholds—and certain derivative models to, among other things:
- Before training the model, implement certain cybersecurity protections, the capability to promptly enact a full shutdown, a written safety and security protocol, and take reasonable care to implement appropriate measures to prevent "critical harms"—defined as mass casualties, at least $500 million in damage, or other comparable harms.
- Before using the model or making it publicly available: assess whether the model is reasonably capable of causing or materially enabling a critical harm; record and retain test results from the assessment; and take reasonable care to implement appropriate safeguards.
- Neither use, unless exclusively for training or evaluation, nor make publicly available a model if there is an unreasonable risk it will cause or materially enable a critical harm.
- Beginning in 2026, annually retain a third-party auditor to perform an independent audit of a developer's compliance with applicable duties.
- Make public and provide to the AG redacted copies of safety and security protocol and auditors' reports. Upon request, provide to the AG unredacted copies of those documents, which are exempt from the California Public Records Act. Submit annual compliance statements to the AG. Report to the AG safety incidents within 72 hours.
- Requires a person operating a computing cluster to implement written policies and procedures to take certain actions when a customer uses compute resources sufficient to train a covered model, including obtaining information to identify the customer.
- Authorizes the AG to bring a civil action for violations of the bill for specified civil penalties and injunctive and declaratory relief.
- Provides whistleblower protections for employees who have reasonable cause to disclose to the AG or Labor Commissioner information regarding a developer's noncompliance with the bill or regarding unreasonably dangerous models.
- Creates the Board of Frontier Models (board) within the Government Operations Agency, consisting of nine members, five appointed by the Governor and four by the Legislature.
- Requires the Government Operations Agency, by 2027, to issue guidance for preventing unreasonable risks and adopt regulations to update the definition of a "covered entity" and establish auditing standards, subject to approval by the board.
- Upon appropriation, establishes in the Government Operations Agency a consortium required to develop a framework for the creation of a public cloud computing cluster to be known as "CalCompute" that advances the development and deployment of AI, as prescribed.
COMMENTS
With rapid advances in AI, industry insiders—often bound by punitive non-disclosure agreements—have sounded the alarm regarding the potential risks posed by these technologies. A group of current and former employees at frontier AI companies recently wrote: "These risks range from the further entrenchment of existing inequalities, to manipulation and misinformation, to the loss of control of autonomous AI systems potentially resulting in human extinction."
This bill seeks to mitigate the risk of catastrophic harms from AI models so advanced that they are not yet known to exist. The bill would require the developers of such models to develop and implement safety and security protocols before initiating training. Developers must also implement the capability to promptly shutdown models. Following training, developers would be required to perform risk assessments on their models and implement reasonable safeguards, subject to third-party auditing, before using or releasing them. If there is an unreasonable risk a model will cause or materially enable a critical harm—mass casualties, at least $500 million in damage, or other comparable harms—developers are prohibited from releasing or using the model.
The bill requires the Government Operations Agency to adopt certain regulations and guidance, subject to approval by the Board of Frontier Models, which would be established under the bill. The bill also adds whistleblower protections; requires operators of computer clusters to implement "know your customer" requirements, including the ability to shut down any resources being used to train an advanced AI model; and lays the groundwork for the creation of a public computing cluster known as "CalCompute." The Attorney General is charged with civil enforcement of the bill's requirements.
This bill has generated a great deal of commentary, consternation, and misconception. To set the record straight: This bills requirements are not onerous. For one, they only impact models that cost over $100 million in computing power to train—this threshold is baked into the definition of "covered model" provided by the bill and cannot be altered except by future legislative action. Secondly, the safety and security protocols developers would be required to implement broadly align with national and international guidelines, as well as procedures these developers already claim to implement. Finally, the bill only requires that developers implement shutdown capabilities for rogue models they themselves control. The open-source community can rest easy knowing the models they download will not contain an immutable kill switch.
This bill does not create a state-sponsored licensing regime for AI; it does not ban the creation and use of AI above a certain compute threshold; and it does not create exorbitant costs for startups seeking to train large models. Performing basic risk assessments and implementing reasonable safeguards before training, using, or releasing powerful, generally-capable models—and prohibiting their use only in extreme cases involving unreasonable risks of mass casualties or massive economic damages—is the bare minimum that Californians should expect of an industry claiming to have their best interests at heart.
According to the Author
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we've seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians' lives, without compromising safety or security.
SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a public AI research cluster that will allow startups, researchers, and community groups to participate in the development of large-scale AI systems. By providing a broad range of stakeholders with access to the AI development process, CalCompute will help align large-scale AI systems with the values and needs of California communities.
Arguments in Support
The Center for AI Safety Action Fund, a co-sponsor of this bill, writes:
If California does not act to establish a clear and sensible governance framework, the safety of its citizens could be imperiled, the nation's security could be seriously harmed, and AI's enormous potential to improve our world could be derailed. Placing sensible guardrails around serious risks that the most powerful systems might pose, while taking significant steps towards leveling the playing field for academics and startups, is the best way to ensure that CA's citizens can realize the immense benefits of this technology.
Encode Justice, a co-sponsor of this bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as "frontier AI models." These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue. These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden's Executive Order on Artificial Intelligence for their significant national security and public safety implications.
Arguments in Opposition
California Chamber of Commerce writes on behalf of a coalition of trade associations:
In addition to creating inconsistencies with federal regulations, the bill demands compliance with various vague and impractical, if not technically infeasible, requirements for which developers will be subject to harsh penalties, including potential criminal liability. We are concerned that the bill regulates AI technology as opposed to its high-risk applications, creates significant regulatory uncertainty and therefore high compliance costs, and poses significant liability risks to developers for failing to foresee and block any harmful use of their models by others – all of which inevitably discourages economic and technological innovation. And while recent amendments take important steps in responding to the open-source community, we remain concerned about the impact of the bill on AI research and development in California and the impact on startups. Overall, the bill still makes AI business too risky in California, particularly given the potential penalties under SB 1047.
FISCAL COMMENTS
According to the Assembly Appropriations Committee,
- Costs (General Fund) to GovOps of an unknown but potentially significant amount to establish and operate the Board, and to issue and update regulations. The bill also requires GovOps to, upon appropriation by the Legislature, establish a consortium and develop a framework for the CalCompute public cloud computing cluster. Although these responsibilities are contingent upon appropriation, they add additional, potentially significant General Fund cost pressure to GovOps.
- Costs (General Fund) to DOJ of an unknown but potentially significant amount to maintain and review developer statements of compliance, and to bring civil enforcement actions. Actual costs will depend in part on the number of violations, the number of actions filed, and the amount of workload associated with each action.
- Costs (Labor and Enforcement Compliance Fund) to the Labor Commissioner of an unknown but amount to receive and act on whistleblower reports.
- Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate violations of the bill and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The fiscal year 2024-25 budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
VOTES
SENATE FLOOR: 32-1-7
YES: Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
NO: Grove
ABS, ABST OR NV: Allen, Alvarado-Gil, Dahle, Jones, Nguyen, Niello, Seyarto
ASM PRIVACY AND CONSUMER PROTECTION: 8-0-3
YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson
ABS, ABST OR NV: Joe Patterson, Dixon, Hoover
ASM JUDICIARY: 9-1-2
YES: Kalra, Bauer-Kahan, Bryan, Connolly, Haney, Maienschein, McKinnor, Pacheco, Reyes
NO: Dixon
ABS, ABST OR NV: Joe Patterson, Sanchez
ASM APPROPRIATIONS: 11-3-1
YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart, Pellerin, Villapudua
NO: Sanchez, Dixon, Ta
ABS, ABST OR NV: Jim Patterson
UPDATED
VERSION: August 19, 2024
CONSULTANT: Josh Tosney /Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0004158
Date of Hearing: August 7, 2024
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
SBPCA Bill Id:SB 1047
Author:(Wiener) – As Amended Ver:July 3, 2024
Policy Committee: | Privacy and Consumer Protection | Vote: | 8 - 0 |
Judiciary | 9 - 1 | ||
|
|
Urgency: No State Mandated Local Program: Yes Reimbursable: No
SUMMARY:
This bill establishes requirements the developer of an artificial intelligence (AI) model trained using a specified level of computing power (“covered model”) must meet before training and releasing the model, establishes a state entity to provide oversight and guidance regarding covered model compliance, provides related whistleblower protections, grants enforcement authority to the Attorney General and Labor Commissioner, and creates CalCompute, a public cloud computing cluster.
Specifically, among other provisions, this bill:
- Establishes risk assessment, safety, security, and testing requirements the developer of a covered AI model must fulfill before training the covered model, using the covered model, or making the covered model available for public or commercial use.
- Requires, beginning January 1, 2028, the developer of a covered model to annually retain a third-party auditor to perform an independent audit of compliance with the requirements of the bill.
- Establishes the Frontier Model Division (FMD) within the Government Operations Agency (GovOps) to, among other responsibilities, annually review certifications from developers, establish an accreditation process for third party auditors of covered models, publish anonymized safety reports, and issue guidance, standards, and best practices necessary to prevent unreasonable risks of covered models and covered model derivatives causing or enabling critical harms.
- Establishes the Board of Frontier Models (Board) within GovOps and makes the Board responsible for directly supervising the FMD.
- Requires the developer of a covered model to submit an annual certification to the FMD certifying the developer’s compliance with the bill’s requirements and providing FMD with specified information about its covered model. Beginning January 1, 2028, a developer must also submit its most recent audit report to FMD as part of this annual certification.
- Requires the developer of a covered model to report each AI safety incident affecting the covered model to the FMD, as specified.
- Requires the FMD, by January 1, 2027, and annually thereafter, to issue regulations to update the definition of a “covered model” to ensure it accurately reflects technological developments and applies to AI models that pose the greatest risk of causing or enabling critical harms.
- Requires a person who operates a computing cluster to implement written policies and procedures when a customer utilizes computer resources sufficient to train a covered model.
- Authorizes the Attorney General to bring a civil action to enforce any provision of the bill, and authorizes the Labor Commissioner to enforce any provision that would constitute a violation of the Labor Code, and specifies applicable penalties.
- Provides whistleblower protections under the California Whistleblower Protection Act for employees of the FMD and employees of developers of covered models.
- Requires the California Department of Technology (CDT) to create a public cloud computing cluster known as CalCompute to study the safe and secure deployment of large-scale AI models and fostering equitable innovation.
FISCAL EFFECT
- Costs (General Fund) to GovOps to establish and operate the FMD and the Board. GovOps reports ongoing annual costs between $5 million and $10 million.
- Costs (General Fund) to CDT to establish and operate CalCompute. In the first year of implementation, CDT reports costs of $508,000 for two temporary positions, $6 million for external consultants, and $1 million for GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $3.7 million for 16 additional positions, $3 million for external consultants, and $700,000 for GenAI talent practices. The bill authorizes CDT to receive private donations, grants, and local funds to support CalCompute, which may offset General Fund costs by an unknown amount.
CDT notes it is challenging to hire GenAI talent in the public sector since salaries are often two or three times higher in the private sector, and private sector employers can offer bonuses, equity, and stock options. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees. GovOps will likely also face these challenges and additional cost pressures.
- Costs (General Fund, Labor and Enforcement Compliance Fund) of an unknown but potentially significant amount to the Department of Justice (DOJ) and Labor Commissioner to enforce violations of the bill. Costs to DOJ may be in the high hundreds of thousands to low millions of dollars annually. Actual costs will depend on the number of violations, the number of actions filed, and the amount of workload associated with each action.
- Costs (Trial Court Trust Fund, General Fund) to the courts to adjudicate violations of the bill and whistleblower cases. Actual costs will depend on the number of violations, the number of actions filed, and the amount of court time needed to resolve each case. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The fiscal year 2024-25 budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
- Minor and absorbable costs to Judicial Council to review and process the model jury instructions recommended by the FMD, as required by the bill.
According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years.
COMMENTS:
- Purpose. This bill is sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
- Background. As the development of AI models has progressed, scholars, policymakers, philosophers, and others have raised concern about the lack of regulation for the most powerful emerging AI models. As discussed thoroughly in the policy committee analyses of this bill, many people inside and outside the industry have encouraged governments to put an anticipatory regulatory framework in place now, despite the considerable uncertainty about the future of AI modeling, to mitigate the risk of future harm resulting from AI models.
This bill seeks to fill this void in California by implementing risk assessment, safety, analysis, cybersecurity, and testing requirements with which developers of the most powerful AI models must comply. Under the bill, the FMD and its governing Board within GovOps are responsible for monitoring compliance with the bill’s requirements, among many other duties outlined in the bill. The Attorney General may file a civil action to enforce any violation of the bill’s provisions, and the Labor Commissioner may enforce any provision of the bill that is also a violation of the Labor Code. The bill contains an initial definition of a “covered model” that determines which models must comply with the bill’s requirements in the short term. The bill requires the FMD to issue regulations by January 2027 to provide an updated definition of “covered model,” and requires the FMD to update the definition annually thereafter to ensure it captures the most powerful AI models that carry the greatest risk of inflicting harm.
This bill also establishes whistleblower protections for employees of the FMD and employees of developers with covered models, with the goal of protecting employees who report violations and other unsafe practices of covered developers. Finally, the bill establishes CalCompute, a public computing cluster, within CDT.
Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081
Date of Hearing: July 2, 2024
ASSEMBLY COMMITTEE ON JUDICIARY
Ash Kalra, Chair
SBPCA Bill Id:SB 1047
Author:(Wiener) – As Amended Ver:June 20, 2024
As Proposed to be Amended
SENATE VOTE: 32-1
SUBJECT: SAFE AND SECURE INNOVATION FOR FRONTIER ARTIFICIAL INTELLIGENCE MODELS ACT
KEY ISSUES:
- SHOULD A DEVELOPER OF AN ARTIFICIAL INTELLIGENCE (AI) “COVERED MODEL” BE REQUIRED TO TAKE SPECIFIC ACTIONS IN ORDER TO MITIGATE THE RISK OF CATASTROPHIC HARMS FROM THEIR MODELS; AND SHOULD CIVIL PENALTIES BE IMPOSED FOR VIOLATIONS OF THESE REQUIREMENTS?
- SHOULD EMPLOYEES OF DEVELOPERS OF COVERED MODELS AND EMPLOYEES OF THE NEWLY CREATED FRONTIER MODEL DIVISION, LIKE ALL PRIVATE AND PUBLIC EMPLOYEES, BE PROTECTED FROM RETALIATION WHEN REPORTING ILLEGAL AND DANGEROUS ACTS AT THEIR WORKPLACES; AND SHOULD THESE WHISTLEBLOWER PROTECTIONS BE SPECIFIC TO AI?
SYNOPSIS
The rapid advancement of AI technologies brings both unprecedented opportunities and significant challenges. The enormous computational power required to develop advanced AI models unfortunately also poses significant risks. As developers push the boundaries of AI capabilities, the potential for unintended consequences grows. Models trained with vast amounts of data and computational resources can develop behaviors that are difficult to predict and control. This unpredictability, coupled with the high stakes involved, underscores the necessity for stringent oversight and safety protocols. The State of California, being a global hub for technology and innovation, has a unique opportunity and responsibility to lead in the regulation and safe development of AI.
Seeking to protect California and humanity from the very serious but not fully understood future harms of Artificial Intelligence, this well-intentioned and ambitious measure, co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice, establishes a scheme of regulatory guardrails and oversight of AI developers and their “covered models.” By creating the Frontier Model Division (Division) within the California Department of Technology, the bill seeks to ensure continuous oversight and enforcement of these protocols. The Division would also oversee third-party audits and annual compliance certifications, enhancing accountability. Additionally, the bill authorizes the Attorney General to take civil action against non-compliant entities, imposing penalties and ordering the deletion of hazardous AI models if necessary. These provisions collectively work to mitigate the risks of AI misuse by non-state actors, such as terrorists and cybercriminals, thereby safeguarding public safety and national security.
The vast majority of these provisions are outside the jurisdiction of this Committee. Relating to this Committee’s jurisdiction, the bill imposes civil penalties for violations of its provisions; provides whistleblower rights under the California Whistleblower Protection Act to employees of the Division and employees of developers of covered models; and requires the Division to provide guidance to the Judicial Council for the future development of jury instructions to be used in civil actions to enforce the bill’s provisions. The author proposes numerous amendments to clarify these aspects of the bill, as well as clarifying amendments regarding the bill’s definition of “critical harm” and auditing requirements. The amendments are included in the SUMMARY, below, and explained in the analysis.
The bill is supported and opposed by dozens of organizations, companies, and academics working in AI and technology. It recently was approved by the Assembly Committee on Privacy and Consumer Protection by a vote of eight to zero.
SUMMARY: Requires a developer of an Artificial Intelligence (AI) “covered model” to take specific actions in order to mitigate the risk of catastrophic harms from such models; imposes civil penalties for violations; and provides whistleblower rights to employees of the Frontier Model Division and employees of developers of covered models. Specifically, this bill:
- Defines the following:
- “Advanced persistent threat” means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
- “Artificial intelligence” means an engineered or machine based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- “Artificial intelligence safety incident” means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
- A covered model autonomously engaging in behavior other than at the request of a user.
- Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model.
- The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model.
- Unauthorized use of a covered model to cause or enable a critical harm.
- “Fine-tuning” to mean adjusting the model weights of a trained model by exposing it to additional data.
- Before Jan. 1, 2027, defines “covered model” to mean either of the following:
- An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds $100 million dollars when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
- An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than 3 times 10^25 integer or floating-point operations.
- On and after Jan. 1, 2027, defines “covered model” to mean either of the following:
- An artificial intelligence model trained using a quantity of computing determined by the Frontier Model Division, the cost of which exceeds $100 million dollars when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
- An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division.
- Defines the following:
- “Post-training modification” to mean modifying the capabilities of a covered model by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
- “Covered model derivative” to mean any of the following:
- An unmodified copy of a covered model.
- A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
- Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding 3 times 10^25 integer or floating-point operations.
- On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Frontier Model Division.
- “Developer” means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, or by fine-tuning an existing model using a sufficient quantity of computing power.
- “Critical harm” means any of the following:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
- Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model providing precise instructions for conducting a cyberattack or series of cyberattacks on critical infrastructure.
- Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that both:
- Acts with limited human oversight, intervention, or supervision.
- Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
- Other grave harms to public safety and security that are of comparable severity to the harms described in i) – iii).
- Clarifies that “critical harm” does not include either of the following:
- Harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible from sources other than a covered model.
- Harms caused or materially enabled by a covered model combined with other software (including other models) if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.
- Provides that on and after January 1, 2026, the dollar amounts specified in 4) shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
- Defines the following terms:
- “Critical infrastructure” means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
- “Frontier Model Division” means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
- “Full shutdown” means the cessation of operation of any of the following:
- The training of a covered model.
- A covered model.
- All covered model derivatives controlled by a developer.
- “Model weight” means a numerical parameter in an artificial intelligence model that is adjusted through training and that helps determine how inputs are transformed into outputs.
- “Open-source artificial intelligence model” means an artificial intelligence model that is made freely available and that may be freely modified and redistributed.
- “Person” means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
- “Reasonable assurance” does not mean full certainty or practical certainty.
- “Safety and security protocol” means documented technical and organizational protocols that meet both of the following criteria:
- The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of covered model derivatives.
- The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developer’s covered model.
- Requires that before a developer initially trains a covered model, they do all of the following:
- Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
- Implement the capability to enact a full shutdown.
- Implement a written safety and security protocol that does all of the following:
- If a developer complies with the safety and security protocol, provides reasonable assurance that the developer will not produce a covered model or covered model derivative that poses an unreasonable risk of causing or enabling a critical harm.
- States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
- Identifies specific tests and test results that would be sufficient to provide reasonable assurance of both of the following:
- That a covered model does not pose an unreasonable risk of causing or enabling a critical harm.
- That covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
- Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
- Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make post-training modifications or create another covered model in a manner that may generate hazardous capabilities.
- Provides sufficient detail for third parties to replicate the testing procedure.
- Describes in detail how the developer will fulfill their obligations under this chapter.
- Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.
- Describes in detail the conditions under which a developer would enact a full shutdown.
- Describes in detail the procedure by which the safety and security protocol may be modified.
- Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, monitoring and reporting on implementation.
- Provide a copy of the safety and security protocol to the Frontier Model Division.
- Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
- If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
- Implement other reasonable measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or enabling critical harms.
- Requires that before using a covered model or covered model derivative, or making a covered model or covered model derivative available for commercial or public use, the developer do all of the following:
- Assess whether the covered model is reasonably capable of causing a critical harm.
- Implement reasonable safeguards to prevent a covered model or associated covered model derivatives from causing a critical harm.
- Ensure, to the extent reasonably possible, that the actions of covered models and covered model derivatives can be attributed to them.
- Prohibits a developer from using a covered model or covered model derivative commercially or publicly, or making a covered model or covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.
- Requires a developer of a covered model to annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section.
- Requires, beginning January 1, 2028, a developer of a covered model to annually retain a third-party auditor that conducts audits consistent with best practices for auditing to perform an independent audit of compliance with the requirements of the bill.
- Requires the auditor to produce an audit report including all of the following:
- A detailed assessment of the developer’s steps to comply with the requirements of this section.
- If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
- A detailed assessment of the developer’s internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
- The signature of the lead auditor certifying the results of the auditor.
- Requires a developer of a covered model to annually submit to the Frontier Model Division a certification under penalty of perjury of compliance with the requirements of the bill signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. This paragraph applies as long as the covered model or any covered model derivatives controlled by the developer remain in commercial or public use, or remain available for commercial or public use.
- Requires, in a certification submitted pursuant to 14), a developer to specify or provide, at a minimum, all of the following:
- The nature and magnitude of critical harms that the covered model or covered model derivatives may reasonably cause or enable, and the outcome of the assessment required by 9a), above.
- An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent the covered model or covered model derivatives from causing or enabling critical harms.
- A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the certification, and the contact information of any third parties relied upon to validate compliance.
- Beginning January 1, 2028, the most recent audit report.
- Requires a developer of a covered model to report each artificial intelligence safety incident affecting the covered model, or any covered model derivatives controlled by the developer, to the Frontier Model Division within 72 hours of the developer learning of the artificial intelligence safety incident, or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
- Requires a developer to submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section no more than 30 days after making a covered model or covered model derivative available for commercial or public use for the first time. A developer need not submit a certification for a covered model derivative if the developer has already submitted a certification for the applicable covered model.
- Provides that in fulfilling their obligations under this chapter, a developer shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
- Requires a person that operates a computing cluster to implement written policies and procedures to do all of the following when a customer utilizes computer resources sufficient to train a covered model:
- Obtain a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
- The identity of that prospective customer.
- The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
- The email address and telephonic contact information used to verify a prospective customer’s identity.
- Assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
- If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to paragraph a) and conduct the assessment required pursuant to paragraph b) prior to each utilization.
- Retain the customer’s internet protocol addresses used to access the cluster, along with the time and date of access or administrative action.
- Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
- Implement the capability to enact a full shutdown of any resources being used to train a covered model.
- Requires a person that operates a computing cluster shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
- Requires a developer of a covered model that provides commercial access to that covered model to provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developer’s terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
- Requires a person that operates a computing cluster to provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developer’s terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
- Provides that a person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
- States that the following are unlawful acts:
- For a developer to fail to comply with any of the requirements of 8) or 21), above.
- For a person who operates a computing cluster to fail to comply with the requirements of 20), 22), or 23), above.
- For a developer to fail to comply with any of the requirements of the whistleblower provisions in 30), below.
- Provides that the following parties may bring a civil action pursuant to 24), above:
- The Attorney General to enforce any provision of the bill.
- The Labor Commissioner to enforce any provision 30), below, that would constitute a violation of the Labor Code.
- Provides that the Attorney General and Labor Commissioner are entitled to recover all of the following in addition to any civil penalties specified in this chapter:
- A civil penalty for a violation that occurs on or after January 1, 2026 in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
- Injunctive or declaratory relief, including but not limited to orders to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer; but provides that the court may only order such relief for a covered model that has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.
- Monetary damages.
- Punitive damages pursuant to standards in existing law. (Civil Code Section 3294 (a).)
- Attorney’s fees and costs.
- Any other relief that the court deems appropriate.
- Declares a provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of the bill, or to shift such a liability to any person or entity in exchange for their use or access of, or right to use or access a developer’s products or services, including by means of a contract of adhesion, is void as a matter of public policy.
- Provides that a court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of to the maximum extent allowed by law if the court concludes that both of the following are true:
- The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
- As the result of the steps described in a), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
- Allows a court to award punitive damages to the prevailing plaintiff in an action pursuant to subdivision where any of the provisions of Civil Code Section 3294 apply.
- Provides that a developer of a covered model or a contractor or subcontractor of the developer shall not do any of the following:
- Prevent, including through terms and conditions of employment, or seek to enforce such terms and conditions of employment, an employee from disclosing information to the Attorney General or the Labor Commissioner, if the employee has reasonable cause to believe either of the following:
- The developer is out of compliance with the requirements of 8), above.
- An artificial intelligence model, including a model that is not a covered model, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
- Retaliate against an employee for disclosing information to the Attorney General or Labor Commissioner, if the employee has reasonable cause to believe either of the above.
- Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 of Division 7 of the Business and Professions Code or any other provision of California law.
- Specifies that an employee harmed by a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
- Specifies that an employee of the Frontier Model Division may report any violation of this chapter by the Frontier Model Division to the State Auditor pursuant to the provisions of the California Whistleblower Protection Act which shall govern any such report.
- Specifies that the Attorney General or Labor Commissioner may publicly release or provide, to the Frontier Model Division or the Governor, any complaint, or a summary of that complaint, pursuant to this section if they conclude that doing so will serve the public interest.
- Requires a developer and any contractor or subcontractor of the developer to provide a clear notice to all employees working on covered models of their rights and responsibilities under this section.
- Provides that a developer is presumed to be in compliance with the requirements of 34) if the developer does one of the following:
- At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice, and ensure that employees who work remotely periodically receive an equivalent notice.
- Provides, no less frequently than once every six months, written notice to all employees of their rights and responsibilities under this chapter and take reasonable steps to ensure that such notice is received and acknowledged by all such employees.
- Requires a developer and any contractor or subcontractor of the developer to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer has violated any provision of the bill or any other law, or has made false or materially misleading statements related to its safety and security protocol or failed to disclose known risks to employees, including, at a minimum, a monthly update to the disclosing employee regarding the status of the employee’s disclosure and the actions taken by the developer, contractor, or subcontractor in response to the disclosure.
- Requires the disclosures and responses of the process required by 36) to be maintained for a minimum of seven years after the date of the disclosure and response and requires the disclosure and responses to be shared with officers and directors of the developer whose acts or omissions are not implicated by the disclosure and any contractor or subcontractor of the developer whose acts or omissions are not implicated by the disclosure at least once per quarter.
- Clarifies that the bill shall not be construed to limit protections provided to employees by specified sections of the Labor Code, the Government Code, or other provisions of California law.
- Defines the following for purposes of 31) – 38), above:
- “Employee” has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
- Contractors or subcontractors, and unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
- Corporate officers.
- “Contractor or subcontractor” has the same meaning as in Section 1777.1 of the Labor Code.
- “Information about non-compliance with the requirements of Section 22603” shall be considered information about unlawful acts in the workplace within the meaning of Section 12964.5 of the Government Code.
- Clarifies that the duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
- Establishes the Board of Frontier Models in the Government Operations Agency. Allows the Governor to appoint an executive director of the Board, subject to Senate confirmation, to exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged. Specifies that the board shall be composed of five members, as follows:
- A member of the open-source community, appointed by the Governor, subject to Senate confirmation.
- A member of the artificial intelligence industry, appointed by the Governor, subject to Senate confirmation.
- A member of academia, appointed by the Governor, subject to Senate confirmation.
- A member appointed by the Speaker of the Assembly.
- A member appointed by the Senate Rules Committee.
- Establishes the Frontier Model Division in the Government Operations Agency, under the direct supervision of the Board. Requires the Division to do the following:
- Annually review certifications received from developers.
- Advise the Attorney General on potential violations of the bill’s provisions.
- Establish an accreditation process for third party auditors.
- Publish anonymized safety reports.
- Issue guidance, standards, and best practices necessary to prevent unreasonable risks of covered models and covered model derivatives causing or enabling critical harms, including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
- Appoint and consult with an advisory committee for open-source AI that shall do all of the following:
- Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that lack the ability to cause or enable critical harms.
- Advise the Legislature on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
- Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
- Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
- Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt consistent with its policies and procedures for the promulgation of jury instructions.
- Requires, in developing the proposed model jury instructions required by 41), the Frontier Model Division shall consider and incorporate all of the following into the proposal that it submits to the Judicial Council:
- All of the actions that a developer of a covered model must take pursuant to Section 22603 of the Business and Professions Code.
- How any regulations of the Frontier Model Division should be incorporated into the proposed model jury instructions.
- The rigor and quality of the safety and security protocol (SSP) that a developer is required to implement while training and releasing its AI model, and how to determine whether this SSP was inferior, comparable, or superior to the SSPs of similarly situated developers.
- The rigor and quality of the developer’s investigation, documentation, evaluation, and management of its model’s potential hazardous capabilities, and associated risks.
- Requires the Frontier Model Division, on or before January 1, 2027, and annually thereafter, to issue regulations to update the definition of a “covered model” to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of causing or enabling critical harms. The updated definition shall contain both of the following:
- The initial compute threshold that an artificial intelligence model must exceed to be considered a covered model, as defined in Section 22602 of the Business and Professions Code.
- The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.
- Requires, in developing regulations, the Frontier Model Division to take into account both of the following:
- The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or enable a critical harm.
- Similar thresholds used in federal law, guidance, or regulations for the management of models with reasonable risks of causing or enabling critical harms.
- Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
- Requires, every 24 months after initial publication of guidance, the Frontier Model Division, to review existing guidance in consideration of technological advancements, changes to industry best practices, and information received to update its guidance to the extent appropriate.
- Requires, on and after January 1, 2026, the Frontier Model Division to annually publish the inflation-adjusted dollar amounts applicable as provided above.
- Tasks the Department of Technology with creating a public cloud computing cluster known as CalCompute through the commissioning of consultants with specified objectives, first of which is to study the safe and secure deployment of large-scale AI models. The consultants shall include representatives of national laboratories, public universities, and any relevant professional associations or private sector stakeholders. They shall evaluate and incorporate the following considerations into their plan:
- An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
- The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
- A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
- An analysis of the state’s investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
- A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
- Authorizes the Department of Technology to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate the establishment of CalCompute.
EXISTING LAW:
- Establishes the Government Operations Agency. (Government Code Section 12800. All further statutory references are to the Government Code, unless otherwise indicated.)
- Establishes the Department of Technology within the Government Operations Agency. (Section 12803.2.)
- Charges the Department of Technology with approving and overseeing information technology projects in the state. (Section 11546.)
- Pursuant to the California Whistleblower Protection Act (CWPA), prohibits “improper governmental activities” by state agencies and employees. (Section 8547.2.)
- Declares that state employees should be free to report waste, fraud, abuse of authority, violation of law, or threat to public health without fear of retribution. (Section 8547.1.)
- Includes the following relevant definitions in the CWPA:
- “Employee” means an individual appointed by the governor, or employed or holding office in a state agency. (Section 8547.2 (a).)
- “State agency” includes every state office, officer, department, division, bureau, board, and commission other than the California State University, unless specifically provided otherwise. (Section 11000 (a).)
- “Protected disclosure” means a good faith communication that discloses or demonstrates an intention to disclose information that may show either (1) an improper governmental activity, or (2) a condition that may threaten the health or safety of employees or the public, if the purpose of disclosing is to remedy that condition. (Section 8547.2 (e).)
- “Improper governmental activity” means an activity by a state agency or by an employee that is undertaken in performance duties, undertaken inside a state office, or, if undertaken outside a state office by the employee, directly relates to state government, whether or not that activity is within the scope of his or her employment, and also meets at least one of the following criteria:
- It is in violation of any state or federal law or regulation, including, but not limited to, corruption, malfeasance, bribery, theft of government property, fraudulent claims, fraud, coercion, conversion, malicious prosecution, misuse of government property, or willful omission to perform a duty.
- It is in violation of an Executive order of the Governor, a California Rule of Court, or any policy or procedure mandated by the State Administrative Manual or State Contracting Manual.
- It is economically wasteful, involves gross misconduct, incompetency, or inefficiency. (Section 8547.2 (c).)
- Requires the State Auditor's Office (SAO), in order to “be free of organizational impairments to independence, to be independent of the executive branch and legislative control” and “to remain free from the influence of existing control agencies that could be the subject of audits conducted by the office.” (Sections 8543, 8546.)
- Ensures the confidentiality of all protected disclosures, including a good faith communication to the SAO alleging an improper governmental activity and any evidence delivered to the SAO in support of the allegation. (Section 8547.2 (e).)
- Requires the SAO to accept complaints by mail and via Internet, and to conduct investigations of alleged improper governmental activities, and authorizes the SAO to issue reports of its findings including recommended corrective actions if it finds reasonable cause to believe an improper governmental activity has occurred. (Sections 8547.4, 8547.5, 8547.7.)
- Allows the SAO to permit complaints to be filed anonymously, in which case the SAO is required to keep the identity of all complainants and witnesses confidential, unless given the express permission of the person, or the disclosure is made to a law enforcement agency that is conducting a criminal investigation. (Section 8547.5 (a).)
- Provides that if the SAO determines that there is reasonable cause to believe that a state agency or employee may have engaged in an improper governmental activity, the SAO may refer the allegation to the involved state agency, or to another state agency having direct oversight of the involved state agency, to conduct an investigation of the allegation under the SAO’s supervision. (Section 8547.6 (b).)
- Allows a state employee or applicant for state employment (including an employee of the SAO or applicant for employment at the SAO) who files a written complaint with his or her supervisor, manager, or the appointing power alleging actual or attempted acts of reprisal, retaliation, threats, coercion, or similar improper acts prohibited by Section 8547.3 to also file a copy of the written complaint with the State Personnel Board (SPB), together with a sworn statement that the contents of the written complaint are true. (Section 8547.8 (a), 8547.13 (b).)
- Provides that if the SPB determines there is a reasonable basis for an alleged violation, it shall transmit a copy of the investigation report to the SAO. (Section 8547.9.)
- Requires the employing state agency to take adverse employment action against any employee found by the SAO to have engaged or participated in improper governmental activity or to set forth in writing its reasons for not taking adverse action, and likewise requires the employing agency to report on actions it has taken to implement the SAO’s recommendations to the SPB and the SAO. (Sections 8547.4.)
- Makes it unlawful for any employer to discharge, expel, or otherwise discriminate against any person because the person has opposed any practices forbidden under the section or because the person has filed a complaint, testified, or assisted in any proceeding; and authorizes the Civil Rights Department to receive, investigate, conciliate, mediate, and prosecute complaints alleging practices made unlawful pursuant to Chapter 6 (commencing with Section 12940). (Sections 12940 (h), 12930 (f)(1).)
- Prohibits employers from preventing an employee from, or retaliate against them for, disclosing information to a government or law enforcement agency, if the employee has reasonable cause to believe that the information discloses a violation of state or federal statute, or a violation of or noncompliance with a local, state, or federal rule or regulation, regardless of whether disclosing the information is part of the employee’s job duties. (Labor Code Section 1102.5.)
- Authorizes, in an action for the breach of an obligation not arising from contract, where it is proven by clear and convincing evidence that the defendant has been guilty of oppression, fraud, or malice, the plaintiff, in addition to the actual damages, may recover damages for the sake of example and by way of punishing the defendant. (Civil Code Section 3294 (a).)
- Defines, for purposes of 17), the following:
- “Malice” means conduct which is intended by the defendant to cause injury to the plaintiff or despicable conduct which is carried on by the defendant with a willful and conscious disregard of the rights or safety of others.
- “Oppression” means despicable conduct that subjects a person to cruel and unjust hardship in conscious disregard of that person’s rights.
- “Fraud” means an intentional misrepresentation, deceit, or concealment of a material fact known to the defendant with the intention on the part of the defendant of thereby depriving a person of property or legal rights or otherwise causing injury. (Civil Code Section 3294 (c).)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS: Seeking to protect California and humanity from the very serious but not fully understood future harms of Artificial Intelligence, this well-intentioned and ambitious measure establishes a scheme of regulatory guardrails and oversight of AI developers and their “covered models.” According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
Background: The promise and peril of AI. The rapid advancement of AI technologies brings both unprecedented opportunities and significant challenges. The enormous computational power required to develop advanced AI models unfortunately also poses significant risks. As developers push the boundaries of AI capabilities, the potential for unintended consequences grows. Models trained with vast amounts of data and computational resources can develop behaviors that are difficult to predict and control. This unpredictability, coupled with the high stakes involved, underscores the necessity for stringent oversight and safety protocols. The State of California, being a global hub for technology and innovation, has a unique opportunity and responsibility to lead in the regulation and safe development of AI.
There have been numerous concerns about AI being used to develop autonomous weapons, which can select and engage targets without human intervention. AI-enabled technologies, such as drones and cyber capabilities, are increasingly accessible to non-state actors due to the proliferation of inexpensive, commercial off-the-shelf AI. These technologies can be used for a variety of harmful purposes, including large-scale misinformation campaigns, cyber espionage, and cyberattacks. For example, non-state actors have used AI to enhance the efficiency of their operations, predict the movements of law enforcement and military personnel, and carry out more sophisticated attacks. (Brookings, "The implications of the AI boom for nonstate armed actors" (April 5, 2023), https://www.brookings.edu/blog/order-from-chaos/2023/04/05/the-implications-of-the-ai-boom-for-nonstate-armed-actors/.)
Specific incidents illustrating these threats include the use of drones by the Islamic State group, which in 2017 deployed a drone to drop an explosive in a residential complex in Iraq. This incident demonstrates the potential for AI-enhanced drones to be used in terrorist attacks. Additionally, drug cartels have employed drones to transport narcotics and deliver bombs, highlighting the dual-use nature of commercial drone technology and the ease with which it can be repurposed for malicious activities. (Brookings, "Cascading chaos: Nonstate actors and AI on the battlefield" (June 2023), https://www.brookings.edu/research/cascading-chaos-nonstate-actors-and-ai-on-the-battlefield/.)
AI-driven algorithms have been used to automate phishing attacks, enhance malware, and find vulnerabilities in critical infrastructure. In 2017, the WannaCry ransomware attack, which utilized AI techniques, affected hundreds of thousands of computers worldwide, causing billions of dollars in damage. WannaCry exploited a vulnerability in Microsoft Windows, encrypting users' data and demanding ransom payments in Bitcoin. This attack disrupted services in various sectors, including healthcare, where the UK’s National Health Service experienced severe operational impacts, and industries such as telecommunications and logistics faced significant disruptions. (BBC, "WannaCry cyber attack: What can we learn?" (May 22, 2017), https://www.bbc.com/news/business-40008343; Brookings, "The implications of the AI boom for nonstate armed actors" (April 5, 2023), https://www.brookings.edu/blog/order-from-chaos/2023/04/05/the-implications-of-the-ai-boom-for-nonstate-armed-actors/; Microsoft, "Staying ahead of threat actors in the age of AI" (May 25, 2023), https://www.microsoft.com/security/blog/2023/05/25/staying-ahead-of-threat-actors-in-the-age-of-ai/.)
The hacktivist group Anonymous has utilized AI-enhanced denial-of-service (DoS) attacks against corporations and right-wing conspiracy theorists, showing the disruptive potential of AI in cyber warfare. These attacks involve overwhelming a target's servers with massive amounts of traffic generated by AI algorithms, causing the servers to crash and the targeted services to become unavailable. By automating the attack process, AI allows for a more efficient and large-scale execution, making it difficult for traditional defense mechanisms to cope. Furthermore, the DarkSide hacker group conducted a significant cyberattack on the Colonial Pipeline in 2021. They used AI-driven malware to breach the pipeline's network security, encrypt data, and demand a ransom for its release. This attack caused widespread disruption to fuel supplies across the Eastern United States, highlighting the vulnerabilities of critical infrastructure to sophisticated AI-powered cyber threats. (Microsoft, "Staying ahead of threat actors in the age of AI" (May 25, 2023), https://www.microsoft.com/security/blog/2023/05/25/staying-ahead-of-threat-actors-in-the-age-of-ai/.)
Automated trading algorithms, if not properly regulated, can lead to market manipulation and financial instability. The 2010 Flash Crash, where AI-driven high-frequency trading algorithms caused a sudden and severe drop in the US stock market, serves as a stark reminder of the economic risks posed by unregulated AI systems. During the Flash Crash, the Dow Jones Industrial Average plunged about 1,000 points within minutes, erasing nearly $1 trillion in market value before recovering shortly after. This incident highlighted the vulnerability of financial markets to AI-driven trading systems and underscored the need for regulatory oversight to prevent such occurrences in the future. (Reuters, "What Caused the 'Flash Crash' Rebound?" (May 11, 2010), https://www.reuters.com/article/us-usa-stocks-flashcrash-idUSTRE64A5R220100511; SEC, "Findings Regarding the Market Events of May 6, 2010" (September 30, 2010), https://www.sec.gov/news/studies/2010/marketevents-report.pdf.)
These examples underscore the real and growing threat posed by non-state actors who may leverage AI technology for nefarious purposes. The accessibility of powerful AI tools to malicious entities can amplify their capabilities, making it easier for them to carry out sophisticated attacks with potentially devastating consequences. This highlights the urgent need for comprehensive oversight and regulatory measures, such as those proposed in this bill, to mitigate these risks and ensure that AI technologies are developed and deployed responsibly. By establishing robust safeguards and monitoring mechanisms, California can lead the way in preventing the misuse of AI and protecting public safety and security. (Brookings, "Democratizing harm: Artificial intelligence in the hands of nonstate actors" (November 2021), https://www.brookings.edu/research/democratizing-harm-artificial-intelligence-in-the-hands-of-nonstate-actors/.)
The bill and proposed amendments. In order to guard against the possible, but grave future harms of AI—whether used to create biological weapons or shut down the electrical grid, for example—the bill, co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice, requires AI developers to implement measures such as cybersecurity protections, safety assessments, and a full shutdown capability to prevent unauthorized access and misuse of AI models. By creating the Frontier Model Division (Division) within the California Department of Technology, the bill seeks to ensure continuous oversight and enforcement of these protocols. The Division would also oversee third-party audits and annual compliance certifications, enhancing accountability. Additionally, the bill authorizes the Attorney General to take civil action against non-compliant entities, imposing penalties and ordering the deletion of hazardous AI models if necessary. These provisions collectively work to mitigate the risks of AI misuse by non-state actors, such as terrorists and cybercriminals, thereby safeguarding public safety and national security.
The vast majority of these provisions are outside the jurisdiction of this Committee. Relating to this Committee’s jurisdiction, the bill imposes civil penalties for violations of its provisions; provides whistleblower rights under the California Whistleblower Protection Act to employees of the Division and employees of developers of covered models; and requires the Division to provide guidance to the Judicial Council for the future development of jury instructions to be used in civil actions to enforce the bill’s provisions.
Civil penalties. The bill in print gives only the Attorney General (AG) the ability to bring a civil action in the event that the AG “finds that a person is violating this chapter.” Therefore, for example, the AG could file a complaint against a developer of a covered model or covered model derivative for failure to comply with the bill’s safety and security protocol, or failure to provide reasonable assurance that the developer will not produce covered models or covered model derivatives that pose an unreasonable risk of causing or enabling a critical harm. For a violation that occurs on or after January 1, 2026, the bill authorizes a civil penalty “in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.” While it may be unclear to a lay person what that means in terms of the amount of damages that could be recovered, presumably it would be a substantial amount. According to information provided by the author:
The average performance of AI models is predictably related to how much computational power, measured in FLOP, was used to train them. Performance on any individual task will tend to go up with FLOP used to train a model, but with less predictability. The FLOP used to train the largest models has rapidly increased over the last few years (doubling every 10 months). This will only continue, because of increasing investment, algorithmic improvements, and hardware improvements. (Footnotes omitted.) Today, AI models (which are all believed to be below the 10^26 FLOP threshold) are not capable of causing the catastrophic harms that this bill is intended to prevent.
The bill in print authorizes the AG to obtain injunctive or declaratory relief, including –in exceptional circumstances—an order “to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer.” The bill provides that the court may only order such relief in a case where the covered model “has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.” The bill authorizes the AG to collect attorney’s fees and costs and any other relief that the court deems appropriate.
The bill requires a court to “disregard corporate formalities and impose joint and several liability on affiliated entities” but does not provide legal standards for how to do so. For example, “[E]vidence of ratification of [agent’s] actions by [defendant] and any other findings made under Civil Code section 3294, subdivision (b), must be made by clear and convincing evidence.” (Barton v. Alexander Hamilton Life Ins. Co. of America (2003) 110 Cal.App.4th 1640, 1644.) The bill in print authorizes punitive damages, but in a somewhat confusing manner.
The proposed author’s amendments (below) significantly revise the enforcement provisions of the bill to make clear which acts constitute a violation of the bill; provide authority for the Labor Commissioner to enforce violations of the bill that also constitute a violation of the Labor Code; and clarify the remedies available to these public civil law enforcement officials.
Although the bill in print (and as proposed to be amended) does not create a private right of action for persons who are personally injured by AI and only provides for AG enforcement, the bill in print speaks to “monetary damages to persons” aggrieved by a violation of the bill’s requirements. However, existing laws and legal theories regarding personal injury and property damage nevertheless would seem flexible enough to allow plaintiffs who are personally harmed by AI to recover damages for their injuries. Therefore, at least in theory, current law should authorize a claim to be filed against a developer who negligently, intentionally, or recklessly causes an injury as a result of the use of their “covered model”.
Author’s amendments—Section 22606 (Enforcement). The author proposes to strike out the text of Section 22606 in its entirety and replace it with the following:
22606. (a) The following are unlawful acts:
- For a developer to fail to comply with any of the requirements of Section 22603 or subdivision (a) of Section 22605.
- For a person who operates a computing cluster to fail to comply with the requirements of Section 22604 or subdivision (b) of Section 22605.
- For a developer to fail to comply with any of the requirements of Section 22607.
- The following parties may bring a civil action pursuant to subdivision (a):
- The Attorney General to enforce any provision of this chapter.
- The Labor Commissioner to enforce any provision of Section 22607 that would constitute a violation of the Labor Code.
(c) The parties listed in subdivision (b) are entitled to recover all of the following in addition to any civil penalties specified in this chapter:
(1) A civil penalty for a violation that occurs on or after January 1, 2026 in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(2) (A) Injunctive or declaratory relief, including but not limited to orders to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer.
(B) The court may only order such relief for a covered model that has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.
(3) (A) Monetary damages.
(B) Punitive damages pursuant to subdivision (a) of Civil Code Section 3294.
(4) Attorney’s fees and costs.
(5) Any other relief that the court deems appropriate.
(d) A provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of this chapter, or to shift such a liability to any person or entity in exchange for their use or access of, or right to use or access, a developer’s products or services, including by means of a contract of adhesion, is void as a matter of public policy.
(e) A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section to the maximum extent allowed by law if the court concludes that both of the following are true:
(A) The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
(B) As the result of the steps described in subparagraph (A), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(f) This section does not limit the application of other laws.
Whistleblower rights. Employees do not surrender their free speech rights because of their employment. The 1st Amendment protects the ability of employees, whether public or private, to speak about employer misconduct in certain circumstances. Public employees, for example, have the right to disclose information about government misconduct that are matters of public concern. (See Garcetti v. Ceballos (2006) 547 U.S. 410, 419, citing Pickering v. Board of Educ. (1968) 391 U.S. 563.) They are also protected from wrongful termination for doing so. (Ostrofe v. H.S. Crocker Co., Inc. (9th Cir. 1982) 670 F.2d 1378, 1383-84 [upholding claim of wrongful discharge of an employee who reported his employer's violation of Clayton Act, despite absence of anti-retaliation clause in the law governing his employment, in order to promote “interests of antitrust enforcement”], aff'd, 740 F.2d 739 (9th Cir. 1984), cert. denied, 469 U.S. 1200 (1985).)
In light of the fact that not all disclosures of government misconduct are necessarily protected by the 1st Amendment (i.e. misconduct that is not a “matter of public concern”), the Legislature has codified the rights of employees to report improper or illegal workplace conduct without fear of retaliation and has enacted a comprehensive legislative scheme for reporting and investigating such complaints. Among others, three comprehensive laws are designed to protect employees and encourage them, in good faith, to report improper or illegal governmental activity so that it can be remediated: (1) The California Whistleblower Protection Act (CWPA); (2) The Fair Employment and Housing Act (FEHA); and (3) Labor Code Section 1102.5.
The CWPA, codified at Sections 8547 et seq., applies to employees of all state agencies, individuals appointed by the governor, or employed by CSU or the courts. (Section 8547.2 (a).) The CWPA states that it is the public policy of the Legislature that “employees should be free to report waste, fraud, abuse of authority, violation of law, or threat to public health without fear of retribution.” (Section 8457.1.) The CWPA also established an extensive process for the State Auditor to investigate reports that a public employee or agency has engaged in improper governmental activity, allowing allegations of improper governmental activity to be submitted in a confidential manner, and authorizing the Auditor to recommend disciplinary action for misconduct, if appropriate. Retaliation against a state employee for making a protected disclosure is prohibited by law. (Section 8547.8 (b).) A state employee or applicant for state employment who believes that he or she has been the victim of retaliation may file a written complaint with either his or her supervisor, manager, or appointing power. (Section 8547.8 (a).) Alternatively, whistleblower complaints can be made to, and investigated by, the State Personnel Board. (Ibid.)
FEHA, codified as Sections 12900 – 12996 prohibits unlawful discrimination in employment and housing. Wrongful termination in violation of FEHA occurs when an employer fires or otherwise retaliates against an employee who, among other things, has “opposed any practices forbidden under this part or because the person has filed a complaint, testified, or assisted in any proceeding under this part.” (Section 12940 (h).)
Labor Code Section 1102.5 prohibits employers, both public and private, from doing either of the following: 1) preventing an employee from disclosing information about an activity reasonably believed to be unlawful to “a government or law enforcement agency, a person with authority over the employee, or any public body conducting an investigation, hearing, or inquiry,” if the employee has reasonable cause to believe that the information discloses a violation of state or federal statute, or a violation of or noncompliance with a local, state, or federal rule or regulation; or 2) retaliating against an employee for reporting such information. (Labor Code Sections 1102.5 (a) and (b).) “Employee” means any person employed by an employer, private or public, including, but not limited to, individuals employed by the state or any subdivision thereof, any county, city, city and county, including any charter city or county, and any school district, community college district, municipal or public corporation, political subdivision, or the University of California. (Labor Code Section 1106.) Under California Labor Code Section 1102.5, if an employer retaliates against a whistleblower, the employer may be required to reinstate the employee’s employment and work benefits, pay lost wages, and take other steps necessary to comply with the law.
Whistleblowers about the dangers of AI. On June 4, 2024, a number of current and former employees at frontier AI companies, such as Open AI and Google DeepMind, wrote an Open Letter, stating that “[W]e believe in the potential of AI technology to deliver unprecedented benefits to humanity. We also understand the serious risks posed by these technologies. These risks range from the further entrenchment of existing inequalities, to manipulation and misinformation, to the loss of control of autonomous AI systems potentially resulting in human extinction.” (See https://righttowarn.ai/.) The former employees point out in the Open Letter that, “there is no effective government oversight of these corporations, current and former employees are among the few people who can hold them accountable to the public. Yet broad confidentiality agreements block us from voicing our concerns, except to the very companies that may be failing to address these issues. Ordinary whistleblower protections are insufficient because they focus on illegal activity, whereas many of the risks we are concerned about are not yet regulated.”
Whistleblower protections under the bill in print. The bill in print reiterates provisions of existing law in terms of clarifying that employees – whether public employees of the Division, or employees of private developers—have the right to report conduct by their employers that violates the requirements of the bill and are protected from retaliation by their employers in response to such reports. The bill in print does not protect the right of employees to report risky behavior that does not yet amount to a violation of the law, however. Given that workers in the AI industry are “publicly calling for more whistleblower protections, specifically citing confidentiality and/or non-disparagement agreements as problematic barriers to ensuring public safety is protected, arguably the scope of the protections for whistleblower complaints in terms of the type of reports that are protected should be more broad than the scope under the bill in print.
Author/Committee amendments – Whistleblower Protections. As proposed to be amended, employees would enjoy more broad protection against retaliation for protected disclosures, including about misconduct that does not yet constitute a violation of the law, but poses a risk of future harm to the public.
22607. (a) Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a A developer of a covered model or a contractor or subcontractor of the developer shall not prevent do either of the following:
- Prevent, including through terms and conditions of employment, or seek to enforce such terms and conditions of employment, an employee from disclosing information to the Attorney General or the Labor Commissioner, if the employee has reasonable cause to believe
that theinformation indicates that theeither of the following:- The developer is out of compliance with the requirements of Section 22603.
- An artificial intelligence model, including a model that is not a covered model, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
(b) Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate
(2) Retaliate against an employee for disclosing information to the Attorney General or Labor Commissioner, if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603 either subparagraph (A) or (B) of paragraph (1).
(3) Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 of Division 7 of the Business and Professions Code or any other provision of California law.
(b) (1) An employee harmed a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
(2) An employee of the Frontier Model Division may report any violation of this chapter by the Frontier Model Division to the State Auditor pursuant to the provisions of the California Whistleblower Protection Act (commencing with Section 8547 of the Government Code) which shall govern any such report.
(c) The Attorney General or Labor Commissioner may publicly release or provide, to the Frontier Model Division or the Governor, any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes they conclude that doing so will serve the public interest.
(d) Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e) (d) Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a A developer and any contractor or subcontractor of the developer shall provide a clear notice to all employees working on covered models of their rights and responsibilities under this section. A developer is presumed to be in compliance with the requirements of this subdivision if the developer does one of the following:
(1) At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice and ensure that employees who work remotely periodically receive an equivalent notice.
(2) No less frequently than once every six months, provide written notice to all employees of their rights and responsibilities under this chapter and ensure that such notice is received and acknowledged by all such employees.
Jury instructions. Anticipating that when civil claims based upon the bill reach the courts, the courts will need technical assistance in the development of jury instructions, the bill tasks the Division with developing proposed model jury instructions and providing them to the Judicial Council. The bill in print is highly specific and technical, requiring the proposed model jury instructions to incorporate all of the following “factors”:
(i) The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii) Whether and to what extent the developer’s safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii) The extent and quality of the developer’s safety and security protocol’s prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant risk of causing or enabling a critical harm.
(iv) Whether and to what extent the developer and its agents complied with the developer’s safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing or enabling a particular harm.
(v) Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
These “factors” may be more specific and technical than necessary. By making the language more flexible, the Division arguably could be more responsive to power, trends, risks and usage of AI when the time comes to provide Judicial Council with the proposed model instructions.
Author’s amendments – Jury Instructions. As proposed to be amended as detailed below, the bill’s modified language (below) would do just that. The author’s amendments would read as follows:
(9) (A) Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt consistent with its policies and procedures for the promulgation of jury instructions.
(B) In developing the proposed model jury instructions required by subparagraph (A), the Frontier Model Division shall consider and incorporate all of the following factors into the proposal that it submits to the Judicial Council:
(i) The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while trained, stored, and released a covered model. All of the actions that a developer of a covered model must take pursuant to Section 22603 of the Business and Professions Code.
(ii) Whether and to what extent the developer’s safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers. How any regulations of the Frontier Model Division should be incorporated into the proposed model jury instructions.
(iii) The extent and quality of the developer’s safety and security protocol’s prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities. The rigor and quality of the safety and security protocol (SSP) that a developer is required to implement while training and releasing its AI model, and how to determine whether this SSP was inferior, comparable, or superior to the SSPs of similarly situated developers.
(iv) Whether and to what extent the developer and its agents complied with the developer’s safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm. The rigor and quality of the developer’s investigation, documentation, evaluation, and management of its model’s potential hazardous capabilities, and associated risks.
(v) Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
Additional author’s amendments—“critical harm” and auditing. The author also proposes a number of clarifying amendments to other provisions of the bill. Among other things, these proposed amendments clarify the following aspects of the bill:
Revised definition of “critical harm.” The bill in print includes confusing language, especially about AI engaging in conduct that a machine cannot commit without a brain or body, such as an act that “would constitute a serious or violent felony under the Penal Code if undertaken by a human with the requisite mental state.” The amendments read as follows:
(B) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure, occurring either in a single incident or over multiple related incidents infrastructure by a model providing precise instructions for conducting a cyberattack or series of cyberattacks on critical infrastructure.
(C) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model autonomously engaging in conduct that would constitute a serious or violent felony under the Penal Code if undertaken by a human with the requisite mental state both:
(i) Acts with limited human oversight, intervention, or supervision.
(ii) Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
(D) Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2) “Critical harm” does not include harms either of the following:
(1) Harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible from sources other than a covered model.
(2) Harms caused or materially enabled by a covered model combined with other software (including other models) if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.
More robust auditing provisions. The bill in print merely requires that, “Beginning January 1, 2028, obtain a certificate of compliance from a third-party auditor who has been accredited pursuant to 11547.6 of the Government Code” but does not specify what information is subject to audit and what powers an auditor has in conducting an audit. The amendments do the following:
- Specify what an audit report should contain in statute and clarifies the role of auditors in the compliance process detailed in Sections 22603:
(a)(4) Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third-party auditors.
. . .
(b)(4) Beginning January 1, 2028, obtain a certificate of compliance from a third-party auditor who has been accredited pursuant to 11547.6 of the Government Code.
. . .
(e) (1) Beginning January 1, 2028, a developer of a covered model shall annually retain a third-party auditor that conducts audits consistent with best practices governing auditors to perform an independent audit of compliance with the requirements of this section.
(2) The auditor shall produce an audit report including all of the following:
(A) A detailed assessment of the developer’s steps to comply with the requirements of this section.
(B) If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
(C) A detailed assessment of the developer’s internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
(D) The signature of the lead auditor certifying the results of the auditor.
. . .
(f)(2)(D) Beginning January 1, 2028, a certificate of compliance from an accredited third-party auditor the most recent audit report pursuant to subdivision (e).
- Clarify the Division’s role in auditing, specifying that it is to issue guidance on auditing best practices rather than manage an accreditation process. (Section 11547.6 (e)(2)(B).)
(B) Establish an accreditation process and relevant accreditation standards under which third-party auditors may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to their requirements under Section 22603 of the Business and Professions Code Issue guidance regarding best practices for conducting an audit pursuant to subdivision (e) of Section 22603 of the Business and Professions Code.
Additional amendments to the bill sought by the groups that would support the bill, if amended. Q8 Empowering Change, a 501(c)3 nonprofit organization dedicated to promoting responsible and inclusive AI development supports the bill, if amended. They raises concerns about the abstract and confusing nature of the regulations for AI developers and the bill's focus on long-term catastrophic outcomes rather than immediate AI-enabled harms. They suggest that the bill should concentrate solely on the creation of CalCompute as a public cloud computing cluster. Additionally, the group recommends ensuring that CalCompute is publicly owned and operated for public benefit, including serving California’s public universities. To prevent the diversion of taxpayer dollars to private entities, they propose explicit language to this effect. The group emphasizes the need for strong whistleblower protections and a private right of action to enhance enforcement.
Additionally, the group focuses on ensuring that CalCompute remains a truly public option, free from the influence of private donations or stakeholders. They express concerns about the limited enforcement capabilities due to scarce budgets and potential conflicts of interest and collusion within the Frontier Model Division. To foster public trust and enhance enforcement, they suggest clarifying CalCompute's public ownership and operation, adding a private right of action and whistleblower protections, and prohibiting conflicts of interest and interlocking directorates among regulated entities. Furthermore, they recommend strengthening "know your customer" (KYC) requirements for computing clusters and implementing ex ante structural separations between cloud services and model services to prevent market abuse.
Oakland Privacy, which also supports the bill, if it is amended, has expressed concerns regarding the definitions of AI safety incidents, which currently exclude those requested by users, potentially leaving unsafe incidents unaddressed. They also highlight the lack of independent third-party review for positive safety determinations (PSDs) and the limitation of enforcement to the State Attorney General, without provision for a private right of action. To address these issues, Oakland Privacy suggests removing the disclaimer in the definition of AI safety incidents, requiring independent third-party review of PSDs, and adding a limited private right of action for individuals harmed by critical safety incidents. Additionally, they recommend that the Frontier Model Division (FMD) provide or coordinate third-party expert witness testimony for civil suits.
Exemptions sought by various industries. The healthcare/life science industry suggests an exemption from the requirement to implement an emergency shutdown switch in AI systems. They argue that AI technologies in healthcare are deeply integrated into essential medical operations, such as diagnostics, treatment planning, and patient monitoring, and an emergency shutdown could result in severe disruptions to patient care. They propose tailored regulations that balance innovation with patient safety to avoid redundancy, increased compliance costs, and deterred innovation in developing new AI applications. (California Life Science.)
The research industry, particularly academic and non-profit research institutions, recommends an exemption from the stringent regulatory requirements of the bill. They argue that broad and prescriptive regulations, including the requirement for emergency shutdown capabilities, could stifle the open exchange of ideas, collaborative experimentation, and rapid development of innovative technologies. They suggest tailored regulations that recognize the unique context of academic research, taking into account existing ethical guidelines and review processes such as Institutional Review Boards (IRBs) and peer review, to support the advancement of AI while maintaining ethical standards. (Q8 Empowering Change, California Life Science.)
Amendments sought by opponents. Focusing on high-risk uses of AI, rather than broadly regulating all AI technologies, is a suggestion made by multiple stakeholders to improve this measure. One primary recommendation is to concentrate regulatory efforts on specific applications of AI that have the potential to cause significant harm if misused, such as autonomous weapons, large-scale surveillance systems, and critical infrastructure controls. (BSA | the Software Alliance.)
Additionally, it has been suggested that the bill should encourage voluntary compliance with safety and security measures for lower-risk AI applications. The proponents of voluntary compliance argue that this approach would promote responsible AI development without imposing mandatory requirements that might be unnecessary for certain applications. Incentivizing voluntary adherence to best practices, according to industry leaders, can foster a culture of safety and ethics within the AI community. In summary, refining this bill to focus on high-risk AI uses involves targeting specific applications, developing a risk-based framework, aligning with industry best practices, and encouraging voluntary compliance for lower-risk applications. According to Abundance Institute, these suggestions aim to balance innovation with safety, ensuring that regulatory efforts are directed where they are most needed while supporting the broader development of AI technologies.
ARGUMENTS IN SUPPORT: Numerous advocacy groups and scholarly groups jointly write the following in support of this bill:
California has become a vibrant hub for artificial intelligence. Universities, startups, and technology companies are using Al to accelerate drug discovery, coordinate wildfire responses, optimize energy consumption, uncover rare minerals that produce clean energy, and enhance creativity. Artificial intelligence has enormous potential to benefit our state and the world. California must act now to ensure that it remains at the forefront of dynamic innovation in AI development.
At the same time, scientists, engineers, and business leaders at the cutting edge of this technology — including the three most cited AI researchers of all time — have repeatedly warned policymakers that failure to take appropriate precautions to prevent irresponsible AI development could have severe consequences for public safety and national security. California must ensure that the small handful of companies developing extremely powerful AI models —including companies explicitly aiming to develop “artificial general intelligence” — take reasonable care to prevent their models from causing very serious harms as they continue to produce models of greater and greater power.
Taken as a whole, SB 1047 strikes a good balance in addressing a variety of crucial areas in AI policy.
The Latino Community Foundation also writes in support:
SB 1047 introduces CalCompute, a publicly owned and operated cloud computing cluster aimed at fostering research and development for the betterment of society. Additionally, it mandates fair and transparent pricing and access to encourage market competition, all while establishing sensible guidelines for developers to ensure the safe and responsible creation of large-scale AI systems.
The technology industry is increasingly dominated by a handful of powerful corporations, resulting in significant financial gains and substantial influence. The rise of AI is exacerbating this trend, further consolidating power within the sector. Without equitable access to AI tools and resources, Latino-led nonprofit organizations, along with Latino small businesses and entrepreneurs, face the threat of being left behind in accessing vital public and private resources to expand their operations and effectively serve communities and engage consumers.
AI has the potential to greatly benefit California and its residents, from improving service delivery to communities to managing wildfires and saving energy. However, this potential remains unrealized if the current market dynamics continue to favor a select few corporations. California has the opportunity and responsibility to ensure that the economic prosperity stemming from the tech boom benefits everyone in the state.
ARGUMENTS IN OPPOSITION: Y Combinator, a technology startup accelerator and venture capital firm, writes the following—on behalf of dozens of AI startups--in opposition to the bill:
We have carefully followed the progression of SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. We acknowledge and appreciate the well-meaning efforts to ensure AI's safe and ethical development, in particular potential for the establishment of CalCompute and language within the bill designed to curb noncompetitive activity. Nevertheless, the bill could inadvertently threaten the vibrancy of California's technology economy and undermine competition. Specifically, we remain most concerned about these aspects of the bill:
1. Liability and regulation that is unusual in its burdens: The responsibility for the misuse of LLMs should rest with those who abuse these tools, not with the developers who create them.
2. Arbitrary Threshold for Regulation: Establishing 1026 FLOPs as a regulatory threshold is problematic. The technology is still evolving, and such specific metrics may not adequately capture the capabilities or risks associated with future models.
3. The requirement of a Kill Switch is possibly a backdoor open source AI ban: While intended to ensure safety, the mandated kill switch could function as a de facto ban on open-source AI development.
4. Vague language could be expanded or reinterpreted later to kill Californian tech: The language is extremely vague. Most importantly, as written, a "frontier model" could plausibly apply to existing software, like Google's search algorithm or any social media
Numerous business groups, including the California Chamber of Commerce, California Manufacturers and Technology Association (CMTA), Civil Justice Association of California (CJAC), Computer and Communications Industry Association (CCIA), Silicon Valley Leadership Group, and TechNet, jointly write the following in opposition to the bill:
SB 1047 creates significant regulatory uncertainty by mandating compliance with novel requirements that rely on standards that are overbroad, vague, and impractical, if not infeasible. While SB 1047 is often interpreted to simply require a risk assessment of models to avoid critical harms, doing so would dramatically misunderstand what the bill does in practice and miscalculate the impact it will have on Californians and the economy—even in the streamlined version. At its core, SB 1047 regulates the development of AI, seeking to keep frontier AI developers from innovating AI models that will result in any kind of foreseeable harm—even harms that would not manifest from the model itself. In doing so, the bill requires developers to comply with incredibly vague, broad, impractical, if not impossible, standards when developing “covered models” and determining whether they can provide reasonable assurance that a covered model does not have a hazardous capability or come close to one, creating significant regulatory uncertainty.
REGISTERED SUPPORT / OPPOSITION:
Support
Ae Studio
Ai Safety Student Team (HARVARD)
Apart Research
California State Council of Service Employees International Union (seiu California)
Cambridge Boston Alignment Initiative
Causative Labs
Center for Ai Safety Action Fund
Chapman University
Civic Ai Security Program
Consumer Watchdog
Denizen
Depict.ai
District Council of Iron Workers of The State of California and Vicinity
Economic Security Project Action
Elicit
Encode Justice
Encultured Ai
Enh Alpha LLC
Far Ai, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone Ai
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Loveable Labs Incorporated
Mit Ai Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panolia Laboratories
Panoplia Laboratories
Paper Farms
Redwood Research
Safe Ai Future
Techequity Collaborative
The Future Society
White Space Marketing Group
One individual
Support if Amended
American Economic Liberties Project
Electronic Frontier Foundation
Oakland Privacy
Opposition
8vdx
Abundance Institute
Acclamation Insurance Management Services
Adacta Labs
Aidy
Alec Action
Algolia
Allied Managed Care
Alloy Automation
American Consumer Institute
Andi
Andreessen Horowitz
Andy Ai
Anneal
Antifragile Research Dba Creatorml
Apriora
Argovox
Association of National Advertisers
Baseline Ai
Bilanc Finance INC.
Brighterway
California Chamber of Commerce
California Fuels and Convenience Alliance
California Land Title Association
California Manufacturers & Technology Association
California Manufacturers and Technology Association
Candoriq
Canonical Ai INC
Center for Data Innovation
Chamber of Progress
Chima
Circle Medical
Citron Labs
Civil Justice Association of California
Clarum
Clerky
Coalition of California Chambers – Orange County
Coalition of Small and Disabled Veteran Businesses
Cointracker
Competitive Enterprise Institute
Computer & Communications Industry Association
Computer and Communications Industry Association
Constructable
Consumer Technology Association
Context Fund
Curtsy
Cyble
Deconvolute Ai
Deferred
Dianahr
Digital First Project
Distill
Docsum
Double
Double Finance
Ecomback
Eden - Workplace Management Software
Edge
Ello
Elythea
Entori, INC
Envelope
Extensional, INC
Fetchflow
Figments Corp
Firebender Corp
Flasher Barricade Association
Flexdesk
Focal INC
Fondo
Gigaml
Gleam
Greptile
Happyrobot INC.
Hazel
Hedgy Labs
Insights Association
International Business Machines Corporation (IBM)
International Center for Law & Economics
Invert INC
Invoke Labs
Jo
K-scale Labs
Keywords Ai INC.
Kontractify, INC
Lightski
Linc Ai
Linum
Listening.com
Local Reg
Los Angeles Area Chamber of Commerce
Los Angeles Business Federation
Los Angeles County Business Federation (BIZ-FED)
Lovecast INC Dba Stably Ai
Lumona
Markprompt
Marr Labs Technologies, INC
Martian Mobility INC. Dba Telematica
Metriport
Miru
Movley
National Federation of Independent Business
National Federation of Independent Business (NFIB)
Octolane Ai
Oneshop
Onward
Openmart
Openpod
Orange Collective
Orange County Business Council
Pabio INC.
Patched Codes, INC
Patika Technology, INC
Patterns
Persist Ai
Phonely
Picnichealth
Playground
Poll Everywhere
Preloop INC.
Query Vary
R Street Institute
Rainforest Qa, INC.
Regology INC
Rejuvenation Technologies INC
Retell Ai
Revamp Ai
Reworkd Ai
Rinsed
Rippling
Risotto
Rocketlit INC.
Rootly
San Diego Regional Chamber of Commerce
San Juan Capistrano Chamber of Commerce
Sela
Sepal Ai
Sideguide Technologies INC (known As Firecrawl)
Silicon Valley Leadership Group
Simplify
Software & Information Industry Association
Software and Information Industry Association
Sorting Robotics
Southwest California Legislative Council
Spoken
Sprx
Squire Ai
Stack Ai
Sterling Road
Storyquest, INC.
Storyworth
Sway Finance
Sync Labs
Synova Life Sciences
Tailor
Taxgpt INC
Technet
Tensor
Tesorio INC
Tracecat
Trainy
Truewind Technologies, INC.
Turbolayer INC. Dba Automatica
Turboprop
Turing Labs
Tusk
Twine
Two Dots
Up Dog
Upkeep
Viva Labs
Walrus Tech INC
Willow
Wordware
Y Combinator
Zapier
Zaymo
Zep Ai
Oppose Unless Amended
Answerdotai, INC
California Life Sciences
Google
Q8 Empowering Change
Concerns
Meta
Analysis Prepared by: Alison Merrilees and Nicole Larson / JUD. / (916) 319-2334
Date of Hearing: July 2, 2024
ASSEMBLY COMMITTEE ON JUDICIARY
Ash Kalra, Chair
SBPCA Bill Id:SB 1047
Author:(Wiener) – As Amended Ver:June 20, 2024
As Proposed to be Amended
SENATE VOTE: 32-1
SUBJECT: SAFE AND SECURE INNOVATION FOR FRONTIER ARTIFICIAL INTELLIGENCE MODELS ACT
KEY ISSUES:
- SHOULD A DEVELOPER OF AN ARTIFICIAL INTELLIGENCE (AI) “COVERED MODEL” BE REQUIRED TO TAKE SPECIFIC ACTIONS IN ORDER TO MITIGATE THE RISK OF CATASTROPHIC HARMS FROM THEIR MODELS; AND SHOULD CIVIL PENALTIES BE IMPOSED FOR VIOLATIONS OF THESE REQUIREMENTS?
- SHOULD EMPLOYEES OF DEVELOPERS OF COVERED MODELS AND EMPLOYEES OF THE NEWLY CREATED FRONTIER MODEL DIVISION, LIKE ALL PRIVATE AND PUBLIC EMPLOYEES, BE PROTECTED FROM RETALIATION WHEN REPORTING ILLEGAL AND DANGEROUS ACTS AT THEIR WORKPLACES; AND SHOULD THESE WHISTLEBLOWER PROTECTIONS BE SPECIFIC TO AI?
SYNOPSIS
The rapid advancement of AI technologies brings both unprecedented opportunities and significant challenges. The enormous computational power required to develop advanced AI models unfortunately also poses significant risks. As developers push the boundaries of AI capabilities, the potential for unintended consequences grows. Models trained with vast amounts of data and computational resources can develop behaviors that are difficult to predict and control. This unpredictability, coupled with the high stakes involved, underscores the necessity for stringent oversight and safety protocols. The State of California, being a global hub for technology and innovation, has a unique opportunity and responsibility to lead in the regulation and safe development of AI.
Seeking to protect California and humanity from the very serious but not fully understood future harms of Artificial Intelligence, this well-intentioned and ambitious measure, co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice, establishes a scheme of regulatory guardrails and oversight of AI developers and their “covered models.” By creating the Frontier Model Division (Division) within the California Department of Technology, the bill seeks to ensure continuous oversight and enforcement of these protocols. The Division would also oversee third-party audits and annual compliance certifications, enhancing accountability. Additionally, the bill authorizes the Attorney General to take civil action against non-compliant entities, imposing penalties and ordering the deletion of hazardous AI models if necessary. These provisions collectively work to mitigate the risks of AI misuse by non-state actors, such as terrorists and cybercriminals, thereby safeguarding public safety and national security.
The vast majority of these provisions are outside the jurisdiction of this Committee. Relating to this Committee’s jurisdiction, the bill imposes civil penalties for violations of its provisions; provides whistleblower rights under the California Whistleblower Protection Act to employees of the Division and employees of developers of covered models; and requires the Division to provide guidance to the Judicial Council for the future development of jury instructions to be used in civil actions to enforce the bill’s provisions. The author proposes numerous amendments to clarify these aspects of the bill, as well as clarifying amendments regarding the bill’s definition of “critical harm” and auditing requirements. The amendments are included in the SUMMARY, below, and explained in the analysis.
The bill is supported and opposed by dozens of organizations, companies, and academics working in AI and technology. It recently was approved by the Assembly Committee on Privacy and Consumer Protection by a vote of eight to zero.
SUMMARY: Requires a developer of an Artificial Intelligence (AI) “covered model” to take specific actions in order to mitigate the risk of catastrophic harms from such models; imposes civil penalties for violations; and provides whistleblower rights to employees of the Frontier Model Division and employees of developers of covered models. Specifically, this bill:
- Defines the following:
- “Advanced persistent threat” means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
- “Artificial intelligence” means an engineered or machine based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- “Artificial intelligence safety incident” means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
- A covered model autonomously engaging in behavior other than at the request of a user.
- Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model.
- The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model.
- Unauthorized use of a covered model to cause or enable a critical harm.
- “Fine-tuning” to mean adjusting the model weights of a trained model by exposing it to additional data.
- Before Jan. 1, 2027, defines “covered model” to mean either of the following:
- An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds $100 million dollars when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
- An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than 3 times 10^25 integer or floating-point operations.
- On and after Jan. 1, 2027, defines “covered model” to mean either of the following:
- An artificial intelligence model trained using a quantity of computing determined by the Frontier Model Division, the cost of which exceeds $100 million dollars when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
- An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division.
- Defines the following:
- “Post-training modification” to mean modifying the capabilities of a covered model by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
- “Covered model derivative” to mean any of the following:
- An unmodified copy of a covered model.
- A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
- Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding 3 times 10^25 integer or floating-point operations.
- On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Frontier Model Division.
- “Developer” means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, or by fine-tuning an existing model using a sufficient quantity of computing power.
- “Critical harm” means any of the following:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
- Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model providing precise instructions for conducting a cyberattack or series of cyberattacks on critical infrastructure.
- Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that both:
- Acts with limited human oversight, intervention, or supervision.
- Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
- Other grave harms to public safety and security that are of comparable severity to the harms described in i) – iii).
- Clarifies that “critical harm” does not include either of the following:
- Harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible from sources other than a covered model.
- Harms caused or materially enabled by a covered model combined with other software (including other models) if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.
- Provides that on and after January 1, 2026, the dollar amounts specified in 4) shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
- Defines the following terms:
- “Critical infrastructure” means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
- “Frontier Model Division” means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
- “Full shutdown” means the cessation of operation of any of the following:
- The training of a covered model.
- A covered model.
- All covered model derivatives controlled by a developer.
- “Model weight” means a numerical parameter in an artificial intelligence model that is adjusted through training and that helps determine how inputs are transformed into outputs.
- “Open-source artificial intelligence model” means an artificial intelligence model that is made freely available and that may be freely modified and redistributed.
- “Person” means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
- “Reasonable assurance” does not mean full certainty or practical certainty.
- “Safety and security protocol” means documented technical and organizational protocols that meet both of the following criteria:
- The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by causing or enabling or potentially causing or enabling the creation of covered model derivatives.
- The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developer’s covered model.
- Requires that before a developer initially trains a covered model, they do all of the following:
- Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, misuse of, or unsafe post-training modifications of, the covered model and all covered model derivatives controlled by the developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
- Implement the capability to enact a full shutdown.
- Implement a written safety and security protocol that does all of the following:
- If a developer complies with the safety and security protocol, provides reasonable assurance that the developer will not produce a covered model or covered model derivative that poses an unreasonable risk of causing or enabling a critical harm.
- States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
- Identifies specific tests and test results that would be sufficient to provide reasonable assurance of both of the following:
- That a covered model does not pose an unreasonable risk of causing or enabling a critical harm.
- That covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
- Describes in detail how the testing procedure assesses the risks associated with post-training modifications.
- Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make post-training modifications or create another covered model in a manner that may generate hazardous capabilities.
- Provides sufficient detail for third parties to replicate the testing procedure.
- Describes in detail how the developer will fulfill their obligations under this chapter.
- Describes in detail how the developer intends to implement the safeguards and requirements referenced in this section.
- Describes in detail the conditions under which a developer would enact a full shutdown.
- Describes in detail the procedure by which the safety and security protocol may be modified.
- Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, monitoring and reporting on implementation.
- Provide a copy of the safety and security protocol to the Frontier Model Division.
- Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
- If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
- Implement other reasonable measures to prevent covered models and covered model derivatives from posing unreasonable risks of causing or enabling critical harms.
- Requires that before using a covered model or covered model derivative, or making a covered model or covered model derivative available for commercial or public use, the developer do all of the following:
- Assess whether the covered model is reasonably capable of causing a critical harm.
- Implement reasonable safeguards to prevent a covered model or associated covered model derivatives from causing a critical harm.
- Ensure, to the extent reasonably possible, that the actions of covered models and covered model derivatives can be attributed to them.
- Prohibits a developer from using a covered model or covered model derivative commercially or publicly, or making a covered model or covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.
- Requires a developer of a covered model to annually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section.
- Requires, beginning January 1, 2028, a developer of a covered model to annually retain a third-party auditor that conducts audits consistent with best practices for auditing to perform an independent audit of compliance with the requirements of the bill.
- Requires the auditor to produce an audit report including all of the following:
- A detailed assessment of the developer’s steps to comply with the requirements of this section.
- If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
- A detailed assessment of the developer’s internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
- The signature of the lead auditor certifying the results of the auditor.
- Requires a developer of a covered model to annually submit to the Frontier Model Division a certification under penalty of perjury of compliance with the requirements of the bill signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. This paragraph applies as long as the covered model or any covered model derivatives controlled by the developer remain in commercial or public use, or remain available for commercial or public use.
- Requires, in a certification submitted pursuant to 14), a developer to specify or provide, at a minimum, all of the following:
- The nature and magnitude of critical harms that the covered model or covered model derivatives may reasonably cause or enable, and the outcome of the assessment required by 9a), above.
- An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent the covered model or covered model derivatives from causing or enabling critical harms.
- A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the certification, and the contact information of any third parties relied upon to validate compliance.
- Beginning January 1, 2028, the most recent audit report.
- Requires a developer of a covered model to report each artificial intelligence safety incident affecting the covered model, or any covered model derivatives controlled by the developer, to the Frontier Model Division within 72 hours of the developer learning of the artificial intelligence safety incident, or within 72 hours of the developer learning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
- Requires a developer to submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section no more than 30 days after making a covered model or covered model derivative available for commercial or public use for the first time. A developer need not submit a certification for a covered model derivative if the developer has already submitted a certification for the applicable covered model.
- Provides that in fulfilling their obligations under this chapter, a developer shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
- Requires a person that operates a computing cluster to implement written policies and procedures to do all of the following when a customer utilizes computer resources sufficient to train a covered model:
- Obtain a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
- The identity of that prospective customer.
- The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
- The email address and telephonic contact information used to verify a prospective customer’s identity.
- Assess whether a prospective customer intends to utilize the computing cluster to train a covered model.
- If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to paragraph a) and conduct the assessment required pursuant to paragraph b) prior to each utilization.
- Retain the customer’s internet protocol addresses used to access the cluster, along with the time and date of access or administrative action.
- Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
- Implement the capability to enact a full shutdown of any resources being used to train a covered model.
- Requires a person that operates a computing cluster shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
- Requires a developer of a covered model that provides commercial access to that covered model to provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developer’s terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
- Requires a person that operates a computing cluster to provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster at a given level of quality and quantity subject to the developer’s terms of service and shall not engage in unlawful discrimination or noncompetitive activity in determining price or access.
- Provides that a person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
- States that the following are unlawful acts:
- For a developer to fail to comply with any of the requirements of 8) or 21), above.
- For a person who operates a computing cluster to fail to comply with the requirements of 20), 22), or 23), above.
- For a developer to fail to comply with any of the requirements of the whistleblower provisions in 30), below.
- Provides that the following parties may bring a civil action pursuant to 24), above:
- The Attorney General to enforce any provision of the bill.
- The Labor Commissioner to enforce any provision 30), below, that would constitute a violation of the Labor Code.
- Provides that the Attorney General and Labor Commissioner are entitled to recover all of the following in addition to any civil penalties specified in this chapter:
- A civil penalty for a violation that occurs on or after January 1, 2026 in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
- Injunctive or declaratory relief, including but not limited to orders to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer; but provides that the court may only order such relief for a covered model that has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.
- Monetary damages.
- Punitive damages pursuant to standards in existing law. (Civil Code Section 3294 (a).)
- Attorney’s fees and costs.
- Any other relief that the court deems appropriate.
- Declares a provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of the bill, or to shift such a liability to any person or entity in exchange for their use or access of, or right to use or access a developer’s products or services, including by means of a contract of adhesion, is void as a matter of public policy.
- Provides that a court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of to the maximum extent allowed by law if the court concludes that both of the following are true:
- The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
- As the result of the steps described in a), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
- Allows a court to award punitive damages to the prevailing plaintiff in an action pursuant to subdivision where any of the provisions of Civil Code Section 3294 apply.
- Provides that a developer of a covered model or a contractor or subcontractor of the developer shall not do any of the following:
- Prevent, including through terms and conditions of employment, or seek to enforce such terms and conditions of employment, an employee from disclosing information to the Attorney General or the Labor Commissioner, if the employee has reasonable cause to believe either of the following:
- The developer is out of compliance with the requirements of 8), above.
- An artificial intelligence model, including a model that is not a covered model, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
- Retaliate against an employee for disclosing information to the Attorney General or Labor Commissioner, if the employee has reasonable cause to believe either of the above.
- Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 of Division 7 of the Business and Professions Code or any other provision of California law.
- Specifies that an employee harmed by a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
- Specifies that an employee of the Frontier Model Division may report any violation of this chapter by the Frontier Model Division to the State Auditor pursuant to the provisions of the California Whistleblower Protection Act which shall govern any such report.
- Specifies that the Attorney General or Labor Commissioner may publicly release or provide, to the Frontier Model Division or the Governor, any complaint, or a summary of that complaint, pursuant to this section if they conclude that doing so will serve the public interest.
- Requires a developer and any contractor or subcontractor of the developer to provide a clear notice to all employees working on covered models of their rights and responsibilities under this section.
- Provides that a developer is presumed to be in compliance with the requirements of 34) if the developer does one of the following:
- At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice, and ensure that employees who work remotely periodically receive an equivalent notice.
- Provides, no less frequently than once every six months, written notice to all employees of their rights and responsibilities under this chapter and take reasonable steps to ensure that such notice is received and acknowledged by all such employees.
- Requires a developer and any contractor or subcontractor of the developer to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer has violated any provision of the bill or any other law, or has made false or materially misleading statements related to its safety and security protocol or failed to disclose known risks to employees, including, at a minimum, a monthly update to the disclosing employee regarding the status of the employee’s disclosure and the actions taken by the developer, contractor, or subcontractor in response to the disclosure.
- Requires the disclosures and responses of the process required by 36) to be maintained for a minimum of seven years after the date of the disclosure and response and requires the disclosure and responses to be shared with officers and directors of the developer whose acts or omissions are not implicated by the disclosure and any contractor or subcontractor of the developer whose acts or omissions are not implicated by the disclosure at least once per quarter.
- Clarifies that the bill shall not be construed to limit protections provided to employees by specified sections of the Labor Code, the Government Code, or other provisions of California law.
- Defines the following for purposes of 31) – 38), above:
- “Employee” has the same meaning as defined in Section 1102.5 of the Labor Code and includes both of the following:
- Contractors or subcontractors, and unpaid advisors involved with assessing, managing, or addressing hazardous capabilities of covered models.
- Corporate officers.
- “Contractor or subcontractor” has the same meaning as in Section 1777.1 of the Labor Code.
- “Information about non-compliance with the requirements of Section 22603” shall be considered information about unlawful acts in the workplace within the meaning of Section 12964.5 of the Government Code.
- Clarifies that the duties and obligations imposed by this chapter are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
- Establishes the Board of Frontier Models in the Government Operations Agency. Allows the Governor to appoint an executive director of the Board, subject to Senate confirmation, to exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged. Specifies that the board shall be composed of five members, as follows:
- A member of the open-source community, appointed by the Governor, subject to Senate confirmation.
- A member of the artificial intelligence industry, appointed by the Governor, subject to Senate confirmation.
- A member of academia, appointed by the Governor, subject to Senate confirmation.
- A member appointed by the Speaker of the Assembly.
- A member appointed by the Senate Rules Committee.
- Establishes the Frontier Model Division in the Government Operations Agency, under the direct supervision of the Board. Requires the Division to do the following:
- Annually review certifications received from developers.
- Advise the Attorney General on potential violations of the bill’s provisions.
- Establish an accreditation process for third party auditors.
- Publish anonymized safety reports.
- Issue guidance, standards, and best practices necessary to prevent unreasonable risks of covered models and covered model derivatives causing or enabling critical harms, including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
- Appoint and consult with an advisory committee for open-source AI that shall do all of the following:
- Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that lack the ability to cause or enable critical harms.
- Advise the Legislature on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
- Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
- Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
- Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt consistent with its policies and procedures for the promulgation of jury instructions.
- Requires, in developing the proposed model jury instructions required by 41), the Frontier Model Division shall consider and incorporate all of the following into the proposal that it submits to the Judicial Council:
- All of the actions that a developer of a covered model must take pursuant to Section 22603 of the Business and Professions Code.
- How any regulations of the Frontier Model Division should be incorporated into the proposed model jury instructions.
- The rigor and quality of the safety and security protocol (SSP) that a developer is required to implement while training and releasing its AI model, and how to determine whether this SSP was inferior, comparable, or superior to the SSPs of similarly situated developers.
- The rigor and quality of the developer’s investigation, documentation, evaluation, and management of its model’s potential hazardous capabilities, and associated risks.
- Requires the Frontier Model Division, on or before January 1, 2027, and annually thereafter, to issue regulations to update the definition of a “covered model” to ensure that it accurately reflects technological developments, scientific literature, and widely accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of causing or enabling critical harms. The updated definition shall contain both of the following:
- The initial compute threshold that an artificial intelligence model must exceed to be considered a covered model, as defined in Section 22602 of the Business and Professions Code.
- The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.
- Requires, in developing regulations, the Frontier Model Division to take into account both of the following:
- The quantity of computing power used to train covered models that have been identified as being reasonably likely to cause or enable a critical harm.
- Similar thresholds used in federal law, guidance, or regulations for the management of models with reasonable risks of causing or enabling critical harms.
- Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
- Requires, every 24 months after initial publication of guidance, the Frontier Model Division, to review existing guidance in consideration of technological advancements, changes to industry best practices, and information received to update its guidance to the extent appropriate.
- Requires, on and after January 1, 2026, the Frontier Model Division to annually publish the inflation-adjusted dollar amounts applicable as provided above.
- Tasks the Department of Technology with creating a public cloud computing cluster known as CalCompute through the commissioning of consultants with specified objectives, first of which is to study the safe and secure deployment of large-scale AI models. The consultants shall include representatives of national laboratories, public universities, and any relevant professional associations or private sector stakeholders. They shall evaluate and incorporate the following considerations into their plan:
- An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
- The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
- A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
- An analysis of the state’s investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
- A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
- Authorizes the Department of Technology to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate the establishment of CalCompute.
EXISTING LAW:
- Establishes the Government Operations Agency. (Government Code Section 12800. All further statutory references are to the Government Code, unless otherwise indicated.)
- Establishes the Department of Technology within the Government Operations Agency. (Section 12803.2.)
- Charges the Department of Technology with approving and overseeing information technology projects in the state. (Section 11546.)
- Pursuant to the California Whistleblower Protection Act (CWPA), prohibits “improper governmental activities” by state agencies and employees. (Section 8547.2.)
- Declares that state employees should be free to report waste, fraud, abuse of authority, violation of law, or threat to public health without fear of retribution. (Section 8547.1.)
- Includes the following relevant definitions in the CWPA:
- “Employee” means an individual appointed by the governor, or employed or holding office in a state agency. (Section 8547.2 (a).)
- “State agency” includes every state office, officer, department, division, bureau, board, and commission other than the California State University, unless specifically provided otherwise. (Section 11000 (a).)
- “Protected disclosure” means a good faith communication that discloses or demonstrates an intention to disclose information that may show either (1) an improper governmental activity, or (2) a condition that may threaten the health or safety of employees or the public, if the purpose of disclosing is to remedy that condition. (Section 8547.2 (e).)
- “Improper governmental activity” means an activity by a state agency or by an employee that is undertaken in performance duties, undertaken inside a state office, or, if undertaken outside a state office by the employee, directly relates to state government, whether or not that activity is within the scope of his or her employment, and also meets at least one of the following criteria:
- It is in violation of any state or federal law or regulation, including, but not limited to, corruption, malfeasance, bribery, theft of government property, fraudulent claims, fraud, coercion, conversion, malicious prosecution, misuse of government property, or willful omission to perform a duty.
- It is in violation of an Executive order of the Governor, a California Rule of Court, or any policy or procedure mandated by the State Administrative Manual or State Contracting Manual.
- It is economically wasteful, involves gross misconduct, incompetency, or inefficiency. (Section 8547.2 (c).)
- Requires the State Auditor's Office (SAO), in order to “be free of organizational impairments to independence, to be independent of the executive branch and legislative control” and “to remain free from the influence of existing control agencies that could be the subject of audits conducted by the office.” (Sections 8543, 8546.)
- Ensures the confidentiality of all protected disclosures, including a good faith communication to the SAO alleging an improper governmental activity and any evidence delivered to the SAO in support of the allegation. (Section 8547.2 (e).)
- Requires the SAO to accept complaints by mail and via Internet, and to conduct investigations of alleged improper governmental activities, and authorizes the SAO to issue reports of its findings including recommended corrective actions if it finds reasonable cause to believe an improper governmental activity has occurred. (Sections 8547.4, 8547.5, 8547.7.)
- Allows the SAO to permit complaints to be filed anonymously, in which case the SAO is required to keep the identity of all complainants and witnesses confidential, unless given the express permission of the person, or the disclosure is made to a law enforcement agency that is conducting a criminal investigation. (Section 8547.5 (a).)
- Provides that if the SAO determines that there is reasonable cause to believe that a state agency or employee may have engaged in an improper governmental activity, the SAO may refer the allegation to the involved state agency, or to another state agency having direct oversight of the involved state agency, to conduct an investigation of the allegation under the SAO’s supervision. (Section 8547.6 (b).)
- Allows a state employee or applicant for state employment (including an employee of the SAO or applicant for employment at the SAO) who files a written complaint with his or her supervisor, manager, or the appointing power alleging actual or attempted acts of reprisal, retaliation, threats, coercion, or similar improper acts prohibited by Section 8547.3 to also file a copy of the written complaint with the State Personnel Board (SPB), together with a sworn statement that the contents of the written complaint are true. (Section 8547.8 (a), 8547.13 (b).)
- Provides that if the SPB determines there is a reasonable basis for an alleged violation, it shall transmit a copy of the investigation report to the SAO. (Section 8547.9.)
- Requires the employing state agency to take adverse employment action against any employee found by the SAO to have engaged or participated in improper governmental activity or to set forth in writing its reasons for not taking adverse action, and likewise requires the employing agency to report on actions it has taken to implement the SAO’s recommendations to the SPB and the SAO. (Sections 8547.4.)
- Makes it unlawful for any employer to discharge, expel, or otherwise discriminate against any person because the person has opposed any practices forbidden under the section or because the person has filed a complaint, testified, or assisted in any proceeding; and authorizes the Civil Rights Department to receive, investigate, conciliate, mediate, and prosecute complaints alleging practices made unlawful pursuant to Chapter 6 (commencing with Section 12940). (Sections 12940 (h), 12930 (f)(1).)
- Prohibits employers from preventing an employee from, or retaliate against them for, disclosing information to a government or law enforcement agency, if the employee has reasonable cause to believe that the information discloses a violation of state or federal statute, or a violation of or noncompliance with a local, state, or federal rule or regulation, regardless of whether disclosing the information is part of the employee’s job duties. (Labor Code Section 1102.5.)
- Authorizes, in an action for the breach of an obligation not arising from contract, where it is proven by clear and convincing evidence that the defendant has been guilty of oppression, fraud, or malice, the plaintiff, in addition to the actual damages, may recover damages for the sake of example and by way of punishing the defendant. (Civil Code Section 3294 (a).)
- Defines, for purposes of 17), the following:
- “Malice” means conduct which is intended by the defendant to cause injury to the plaintiff or despicable conduct which is carried on by the defendant with a willful and conscious disregard of the rights or safety of others.
- “Oppression” means despicable conduct that subjects a person to cruel and unjust hardship in conscious disregard of that person’s rights.
- “Fraud” means an intentional misrepresentation, deceit, or concealment of a material fact known to the defendant with the intention on the part of the defendant of thereby depriving a person of property or legal rights or otherwise causing injury. (Civil Code Section 3294 (c).)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS: Seeking to protect California and humanity from the very serious but not fully understood future harms of Artificial Intelligence, this well-intentioned and ambitious measure establishes a scheme of regulatory guardrails and oversight of AI developers and their “covered models.” According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
Background: The promise and peril of AI. The rapid advancement of AI technologies brings both unprecedented opportunities and significant challenges. The enormous computational power required to develop advanced AI models unfortunately also poses significant risks. As developers push the boundaries of AI capabilities, the potential for unintended consequences grows. Models trained with vast amounts of data and computational resources can develop behaviors that are difficult to predict and control. This unpredictability, coupled with the high stakes involved, underscores the necessity for stringent oversight and safety protocols. The State of California, being a global hub for technology and innovation, has a unique opportunity and responsibility to lead in the regulation and safe development of AI.
There have been numerous concerns about AI being used to develop autonomous weapons, which can select and engage targets without human intervention. AI-enabled technologies, such as drones and cyber capabilities, are increasingly accessible to non-state actors due to the proliferation of inexpensive, commercial off-the-shelf AI. These technologies can be used for a variety of harmful purposes, including large-scale misinformation campaigns, cyber espionage, and cyberattacks. For example, non-state actors have used AI to enhance the efficiency of their operations, predict the movements of law enforcement and military personnel, and carry out more sophisticated attacks. (Brookings, "The implications of the AI boom for nonstate armed actors" (April 5, 2023), https://www.brookings.edu/blog/order-from-chaos/2023/04/05/the-implications-of-the-ai-boom-for-nonstate-armed-actors/.)
Specific incidents illustrating these threats include the use of drones by the Islamic State group, which in 2017 deployed a drone to drop an explosive in a residential complex in Iraq. This incident demonstrates the potential for AI-enhanced drones to be used in terrorist attacks. Additionally, drug cartels have employed drones to transport narcotics and deliver bombs, highlighting the dual-use nature of commercial drone technology and the ease with which it can be repurposed for malicious activities. (Brookings, "Cascading chaos: Nonstate actors and AI on the battlefield" (June 2023), https://www.brookings.edu/research/cascading-chaos-nonstate-actors-and-ai-on-the-battlefield/.)
AI-driven algorithms have been used to automate phishing attacks, enhance malware, and find vulnerabilities in critical infrastructure. In 2017, the WannaCry ransomware attack, which utilized AI techniques, affected hundreds of thousands of computers worldwide, causing billions of dollars in damage. WannaCry exploited a vulnerability in Microsoft Windows, encrypting users' data and demanding ransom payments in Bitcoin. This attack disrupted services in various sectors, including healthcare, where the UK’s National Health Service experienced severe operational impacts, and industries such as telecommunications and logistics faced significant disruptions. (BBC, "WannaCry cyber attack: What can we learn?" (May 22, 2017), https://www.bbc.com/news/business-40008343; Brookings, "The implications of the AI boom for nonstate armed actors" (April 5, 2023), https://www.brookings.edu/blog/order-from-chaos/2023/04/05/the-implications-of-the-ai-boom-for-nonstate-armed-actors/; Microsoft, "Staying ahead of threat actors in the age of AI" (May 25, 2023), https://www.microsoft.com/security/blog/2023/05/25/staying-ahead-of-threat-actors-in-the-age-of-ai/.)
The hacktivist group Anonymous has utilized AI-enhanced denial-of-service (DoS) attacks against corporations and right-wing conspiracy theorists, showing the disruptive potential of AI in cyber warfare. These attacks involve overwhelming a target's servers with massive amounts of traffic generated by AI algorithms, causing the servers to crash and the targeted services to become unavailable. By automating the attack process, AI allows for a more efficient and large-scale execution, making it difficult for traditional defense mechanisms to cope. Furthermore, the DarkSide hacker group conducted a significant cyberattack on the Colonial Pipeline in 2021. They used AI-driven malware to breach the pipeline's network security, encrypt data, and demand a ransom for its release. This attack caused widespread disruption to fuel supplies across the Eastern United States, highlighting the vulnerabilities of critical infrastructure to sophisticated AI-powered cyber threats. (Microsoft, "Staying ahead of threat actors in the age of AI" (May 25, 2023), https://www.microsoft.com/security/blog/2023/05/25/staying-ahead-of-threat-actors-in-the-age-of-ai/.)
Automated trading algorithms, if not properly regulated, can lead to market manipulation and financial instability. The 2010 Flash Crash, where AI-driven high-frequency trading algorithms caused a sudden and severe drop in the US stock market, serves as a stark reminder of the economic risks posed by unregulated AI systems. During the Flash Crash, the Dow Jones Industrial Average plunged about 1,000 points within minutes, erasing nearly $1 trillion in market value before recovering shortly after. This incident highlighted the vulnerability of financial markets to AI-driven trading systems and underscored the need for regulatory oversight to prevent such occurrences in the future. (Reuters, "What Caused the 'Flash Crash' Rebound?" (May 11, 2010), https://www.reuters.com/article/us-usa-stocks-flashcrash-idUSTRE64A5R220100511; SEC, "Findings Regarding the Market Events of May 6, 2010" (September 30, 2010), https://www.sec.gov/news/studies/2010/marketevents-report.pdf.)
These examples underscore the real and growing threat posed by non-state actors who may leverage AI technology for nefarious purposes. The accessibility of powerful AI tools to malicious entities can amplify their capabilities, making it easier for them to carry out sophisticated attacks with potentially devastating consequences. This highlights the urgent need for comprehensive oversight and regulatory measures, such as those proposed in this bill, to mitigate these risks and ensure that AI technologies are developed and deployed responsibly. By establishing robust safeguards and monitoring mechanisms, California can lead the way in preventing the misuse of AI and protecting public safety and security. (Brookings, "Democratizing harm: Artificial intelligence in the hands of nonstate actors" (November 2021), https://www.brookings.edu/research/democratizing-harm-artificial-intelligence-in-the-hands-of-nonstate-actors/.)
The bill and proposed amendments. In order to guard against the possible, but grave future harms of AI—whether used to create biological weapons or shut down the electrical grid, for example—the bill, co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice, requires AI developers to implement measures such as cybersecurity protections, safety assessments, and a full shutdown capability to prevent unauthorized access and misuse of AI models. By creating the Frontier Model Division (Division) within the California Department of Technology, the bill seeks to ensure continuous oversight and enforcement of these protocols. The Division would also oversee third-party audits and annual compliance certifications, enhancing accountability. Additionally, the bill authorizes the Attorney General to take civil action against non-compliant entities, imposing penalties and ordering the deletion of hazardous AI models if necessary. These provisions collectively work to mitigate the risks of AI misuse by non-state actors, such as terrorists and cybercriminals, thereby safeguarding public safety and national security.
The vast majority of these provisions are outside the jurisdiction of this Committee. Relating to this Committee’s jurisdiction, the bill imposes civil penalties for violations of its provisions; provides whistleblower rights under the California Whistleblower Protection Act to employees of the Division and employees of developers of covered models; and requires the Division to provide guidance to the Judicial Council for the future development of jury instructions to be used in civil actions to enforce the bill’s provisions.
Civil penalties. The bill in print gives only the Attorney General (AG) the ability to bring a civil action in the event that the AG “finds that a person is violating this chapter.” Therefore, for example, the AG could file a complaint against a developer of a covered model or covered model derivative for failure to comply with the bill’s safety and security protocol, or failure to provide reasonable assurance that the developer will not produce covered models or covered model derivatives that pose an unreasonable risk of causing or enabling a critical harm. For a violation that occurs on or after January 1, 2026, the bill authorizes a civil penalty “in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.” While it may be unclear to a lay person what that means in terms of the amount of damages that could be recovered, presumably it would be a substantial amount. According to information provided by the author:
The average performance of AI models is predictably related to how much computational power, measured in FLOP, was used to train them. Performance on any individual task will tend to go up with FLOP used to train a model, but with less predictability. The FLOP used to train the largest models has rapidly increased over the last few years (doubling every 10 months). This will only continue, because of increasing investment, algorithmic improvements, and hardware improvements. (Footnotes omitted.) Today, AI models (which are all believed to be below the 10^26 FLOP threshold) are not capable of causing the catastrophic harms that this bill is intended to prevent.
The bill in print authorizes the AG to obtain injunctive or declaratory relief, including –in exceptional circumstances—an order “to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer.” The bill provides that the court may only order such relief in a case where the covered model “has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.” The bill authorizes the AG to collect attorney’s fees and costs and any other relief that the court deems appropriate.
The bill requires a court to “disregard corporate formalities and impose joint and several liability on affiliated entities” but does not provide legal standards for how to do so. For example, “[E]vidence of ratification of [agent’s] actions by [defendant] and any other findings made under Civil Code section 3294, subdivision (b), must be made by clear and convincing evidence.” (Barton v. Alexander Hamilton Life Ins. Co. of America (2003) 110 Cal.App.4th 1640, 1644.) The bill in print authorizes punitive damages, but in a somewhat confusing manner.
The proposed author’s amendments (below) significantly revise the enforcement provisions of the bill to make clear which acts constitute a violation of the bill; provide authority for the Labor Commissioner to enforce violations of the bill that also constitute a violation of the Labor Code; and clarify the remedies available to these public civil law enforcement officials.
Although the bill in print (and as proposed to be amended) does not create a private right of action for persons who are personally injured by AI and only provides for AG enforcement, the bill in print speaks to “monetary damages to persons” aggrieved by a violation of the bill’s requirements. However, existing laws and legal theories regarding personal injury and property damage nevertheless would seem flexible enough to allow plaintiffs who are personally harmed by AI to recover damages for their injuries. Therefore, at least in theory, current law should authorize a claim to be filed against a developer who negligently, intentionally, or recklessly causes an injury as a result of the use of their “covered model”.
Author’s amendments—Section 22606 (Enforcement). The author proposes to strike out the text of Section 22606 in its entirety and replace it with the following:
22606. (a) The following are unlawful acts:
- For a developer to fail to comply with any of the requirements of Section 22603 or subdivision (a) of Section 22605.
- For a person who operates a computing cluster to fail to comply with the requirements of Section 22604 or subdivision (b) of Section 22605.
- For a developer to fail to comply with any of the requirements of Section 22607.
- The following parties may bring a civil action pursuant to subdivision (a):
- The Attorney General to enforce any provision of this chapter.
- The Labor Commissioner to enforce any provision of Section 22607 that would constitute a violation of the Labor Code.
(c) The parties listed in subdivision (b) are entitled to recover all of the following in addition to any civil penalties specified in this chapter:
(1) A civil penalty for a violation that occurs on or after January 1, 2026 in an amount not exceeding 10 percent of the cost of the quantity of computing power used to train the covered model to be calculated using average market prices of cloud compute at the time of training for a first violation and in an amount not exceeding 30 percent of that value for any subsequent violation.
(2) (A) Injunctive or declaratory relief, including but not limited to orders to modify, implement a full shutdown, or delete the covered model and any covered model derivatives controlled by the developer.
(B) The court may only order such relief for a covered model that has caused death or bodily harm to another human, harm to property, theft or misappropriation of property, or constitutes an imminent risk or threat to public safety.
(3) (A) Monetary damages.
(B) Punitive damages pursuant to subdivision (a) of Civil Code Section 3294.
(4) Attorney’s fees and costs.
(5) Any other relief that the court deems appropriate.
(d) A provision within a contract or agreement that seeks to waive, preclude, or burden the enforcement of a liability arising from a violation of this chapter, or to shift such a liability to any person or entity in exchange for their use or access of, or right to use or access, a developer’s products or services, including by means of a contract of adhesion, is void as a matter of public policy.
(e) A court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this section to the maximum extent allowed by law if the court concludes that both of the following are true:
(A) The affiliated entities, in the development of the corporate structure among the affiliated entities, took steps to purposely and unreasonably limit or avoid liability.
(B) As the result of the steps described in subparagraph (A), the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
(f) This section does not limit the application of other laws.
Whistleblower rights. Employees do not surrender their free speech rights because of their employment. The 1st Amendment protects the ability of employees, whether public or private, to speak about employer misconduct in certain circumstances. Public employees, for example, have the right to disclose information about government misconduct that are matters of public concern. (See Garcetti v. Ceballos (2006) 547 U.S. 410, 419, citing Pickering v. Board of Educ. (1968) 391 U.S. 563.) They are also protected from wrongful termination for doing so. (Ostrofe v. H.S. Crocker Co., Inc. (9th Cir. 1982) 670 F.2d 1378, 1383-84 [upholding claim of wrongful discharge of an employee who reported his employer's violation of Clayton Act, despite absence of anti-retaliation clause in the law governing his employment, in order to promote “interests of antitrust enforcement”], aff'd, 740 F.2d 739 (9th Cir. 1984), cert. denied, 469 U.S. 1200 (1985).)
In light of the fact that not all disclosures of government misconduct are necessarily protected by the 1st Amendment (i.e. misconduct that is not a “matter of public concern”), the Legislature has codified the rights of employees to report improper or illegal workplace conduct without fear of retaliation and has enacted a comprehensive legislative scheme for reporting and investigating such complaints. Among others, three comprehensive laws are designed to protect employees and encourage them, in good faith, to report improper or illegal governmental activity so that it can be remediated: (1) The California Whistleblower Protection Act (CWPA); (2) The Fair Employment and Housing Act (FEHA); and (3) Labor Code Section 1102.5.
The CWPA, codified at Sections 8547 et seq., applies to employees of all state agencies, individuals appointed by the governor, or employed by CSU or the courts. (Section 8547.2 (a).) The CWPA states that it is the public policy of the Legislature that “employees should be free to report waste, fraud, abuse of authority, violation of law, or threat to public health without fear of retribution.” (Section 8457.1.) The CWPA also established an extensive process for the State Auditor to investigate reports that a public employee or agency has engaged in improper governmental activity, allowing allegations of improper governmental activity to be submitted in a confidential manner, and authorizing the Auditor to recommend disciplinary action for misconduct, if appropriate. Retaliation against a state employee for making a protected disclosure is prohibited by law. (Section 8547.8 (b).) A state employee or applicant for state employment who believes that he or she has been the victim of retaliation may file a written complaint with either his or her supervisor, manager, or appointing power. (Section 8547.8 (a).) Alternatively, whistleblower complaints can be made to, and investigated by, the State Personnel Board. (Ibid.)
FEHA, codified as Sections 12900 – 12996 prohibits unlawful discrimination in employment and housing. Wrongful termination in violation of FEHA occurs when an employer fires or otherwise retaliates against an employee who, among other things, has “opposed any practices forbidden under this part or because the person has filed a complaint, testified, or assisted in any proceeding under this part.” (Section 12940 (h).)
Labor Code Section 1102.5 prohibits employers, both public and private, from doing either of the following: 1) preventing an employee from disclosing information about an activity reasonably believed to be unlawful to “a government or law enforcement agency, a person with authority over the employee, or any public body conducting an investigation, hearing, or inquiry,” if the employee has reasonable cause to believe that the information discloses a violation of state or federal statute, or a violation of or noncompliance with a local, state, or federal rule or regulation; or 2) retaliating against an employee for reporting such information. (Labor Code Sections 1102.5 (a) and (b).) “Employee” means any person employed by an employer, private or public, including, but not limited to, individuals employed by the state or any subdivision thereof, any county, city, city and county, including any charter city or county, and any school district, community college district, municipal or public corporation, political subdivision, or the University of California. (Labor Code Section 1106.) Under California Labor Code Section 1102.5, if an employer retaliates against a whistleblower, the employer may be required to reinstate the employee’s employment and work benefits, pay lost wages, and take other steps necessary to comply with the law.
Whistleblowers about the dangers of AI. On June 4, 2024, a number of current and former employees at frontier AI companies, such as Open AI and Google DeepMind, wrote an Open Letter, stating that “[W]e believe in the potential of AI technology to deliver unprecedented benefits to humanity. We also understand the serious risks posed by these technologies. These risks range from the further entrenchment of existing inequalities, to manipulation and misinformation, to the loss of control of autonomous AI systems potentially resulting in human extinction.” (See https://righttowarn.ai/.) The former employees point out in the Open Letter that, “there is no effective government oversight of these corporations, current and former employees are among the few people who can hold them accountable to the public. Yet broad confidentiality agreements block us from voicing our concerns, except to the very companies that may be failing to address these issues. Ordinary whistleblower protections are insufficient because they focus on illegal activity, whereas many of the risks we are concerned about are not yet regulated.”
Whistleblower protections under the bill in print. The bill in print reiterates provisions of existing law in terms of clarifying that employees – whether public employees of the Division, or employees of private developers—have the right to report conduct by their employers that violates the requirements of the bill and are protected from retaliation by their employers in response to such reports. The bill in print does not protect the right of employees to report risky behavior that does not yet amount to a violation of the law, however. Given that workers in the AI industry are “publicly calling for more whistleblower protections, specifically citing confidentiality and/or non-disparagement agreements as problematic barriers to ensuring public safety is protected, arguably the scope of the protections for whistleblower complaints in terms of the type of reports that are protected should be more broad than the scope under the bill in print.
Author/Committee amendments – Whistleblower Protections. As proposed to be amended, employees would enjoy more broad protection against retaliation for protected disclosures, including about misconduct that does not yet constitute a violation of the law, but poses a risk of future harm to the public.
22607. (a) Pursuant to subdivision (a) of Section 1102.5 of the Labor Code, a A developer of a covered model or a contractor or subcontractor of the developer shall not prevent do either of the following:
- Prevent, including through terms and conditions of employment, or seek to enforce such terms and conditions of employment, an employee from disclosing information to the Attorney General or the Labor Commissioner, if the employee has reasonable cause to believe
that theinformation indicates that theeither of the following:- The developer is out of compliance with the requirements of Section 22603.
- An artificial intelligence model, including a model that is not a covered model, poses an unreasonable risk of causing or materially enabling critical harm, even if the employer is not out of compliance with any law.
(b) Pursuant to subdivision (b) of Section 1102.5 of the Labor Code, a developer shall not retaliate
(2) Retaliate against an employee for disclosing information to the Attorney General or Labor Commissioner, if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of Section 22603 either subparagraph (A) or (B) of paragraph (1).
(3) Make false or materially misleading statements related to its safety and security protocol in a manner that violates Part 2 of Division 7 of the Business and Professions Code or any other provision of California law.
(b) (1) An employee harmed a violation of this subdivision may petition a court for appropriate temporary or preliminary injunctive relief as provided in Sections 1102.61 and 1102.62 of the Labor Code.
(2) An employee of the Frontier Model Division may report any violation of this chapter by the Frontier Model Division to the State Auditor pursuant to the provisions of the California Whistleblower Protection Act (commencing with Section 8547 of the Government Code) which shall govern any such report.
(c) The Attorney General or Labor Commissioner may publicly release or provide, to the Frontier Model Division or the Governor, any complaint, or a summary of that complaint, pursuant to this section if the Attorney General concludes they conclude that doing so will serve the public interest.
(d) Employees shall seek relief for violations of subdivisions (a) and (b) pursuant to Sections 1102.61 and 1102.62 of the Labor Code.
(e) (d) Pursuant to subdivision (a) of Section 1102.8 of the Labor Code, a A developer and any contractor or subcontractor of the developer shall provide a clear notice to all employees working on covered models of their rights and responsibilities under this section. A developer is presumed to be in compliance with the requirements of this subdivision if the developer does one of the following:
(1) At all times post and display within all workplaces maintained by the developer a notice to all employees of their rights and responsibilities under this section, ensure that all new employees receive equivalent notice and ensure that employees who work remotely periodically receive an equivalent notice.
(2) No less frequently than once every six months, provide written notice to all employees of their rights and responsibilities under this chapter and ensure that such notice is received and acknowledged by all such employees.
Jury instructions. Anticipating that when civil claims based upon the bill reach the courts, the courts will need technical assistance in the development of jury instructions, the bill tasks the Division with developing proposed model jury instructions and providing them to the Judicial Council. The bill in print is highly specific and technical, requiring the proposed model jury instructions to incorporate all of the following “factors”:
(i) The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii) Whether and to what extent the developer’s safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii) The extent and quality of the developer’s safety and security protocol’s prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant risk of causing or enabling a critical harm.
(iv) Whether and to what extent the developer and its agents complied with the developer’s safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing or enabling a particular harm.
(v) Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
These “factors” may be more specific and technical than necessary. By making the language more flexible, the Division arguably could be more responsive to power, trends, risks and usage of AI when the time comes to provide Judicial Council with the proposed model instructions.
Author’s amendments – Jury Instructions. As proposed to be amended as detailed below, the bill’s modified language (below) would do just that. The author’s amendments would read as follows:
(9) (A) Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt consistent with its policies and procedures for the promulgation of jury instructions.
(B) In developing the proposed model jury instructions required by subparagraph (A), the Frontier Model Division shall consider and incorporate all of the following factors into the proposal that it submits to the Judicial Council:
(i) The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while trained, stored, and released a covered model. All of the actions that a developer of a covered model must take pursuant to Section 22603 of the Business and Professions Code.
(ii) Whether and to what extent the developer’s safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers. How any regulations of the Frontier Model Division should be incorporated into the proposed model jury instructions.
(iii) The extent and quality of the developer’s safety and security protocol’s prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilities. The rigor and quality of the safety and security protocol (SSP) that a developer is required to implement while training and releasing its AI model, and how to determine whether this SSP was inferior, comparable, or superior to the SSPs of similarly situated developers.
(iv) Whether and to what extent the developer and its agents complied with the developer’s safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm. The rigor and quality of the developer’s investigation, documentation, evaluation, and management of its model’s potential hazardous capabilities, and associated risks.
(v) Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
Additional author’s amendments—“critical harm” and auditing. The author also proposes a number of clarifying amendments to other provisions of the bill. Among other things, these proposed amendments clarify the following aspects of the bill:
Revised definition of “critical harm.” The bill in print includes confusing language, especially about AI engaging in conduct that a machine cannot commit without a brain or body, such as an act that “would constitute a serious or violent felony under the Penal Code if undertaken by a human with the requisite mental state.” The amendments read as follows:
(B) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure, occurring either in a single incident or over multiple related incidents infrastructure by a model providing precise instructions for conducting a cyberattack or series of cyberattacks on critical infrastructure.
(C) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model autonomously engaging in conduct that would constitute a serious or violent felony under the Penal Code if undertaken by a human with the requisite mental state both:
(i) Acts with limited human oversight, intervention, or supervision.
(ii) Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.
(D) Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.
(2) “Critical harm” does not include harms either of the following:
(1) Harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible from sources other than a covered model.
(2) Harms caused or materially enabled by a covered model combined with other software (including other models) if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.
More robust auditing provisions. The bill in print merely requires that, “Beginning January 1, 2028, obtain a certificate of compliance from a third-party auditor who has been accredited pursuant to 11547.6 of the Government Code” but does not specify what information is subject to audit and what powers an auditor has in conducting an audit. The amendments do the following:
- Specify what an audit report should contain in statute and clarifies the role of auditors in the compliance process detailed in Sections 22603:
(a)(4) Ensure that the safety and security protocol is implemented as written, including by designating senior personnel to be responsible for ensuring compliance by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third-party auditors.
. . .
(b)(4) Beginning January 1, 2028, obtain a certificate of compliance from a third-party auditor who has been accredited pursuant to 11547.6 of the Government Code.
. . .
(e) (1) Beginning January 1, 2028, a developer of a covered model shall annually retain a third-party auditor that conducts audits consistent with best practices governing auditors to perform an independent audit of compliance with the requirements of this section.
(2) The auditor shall produce an audit report including all of the following:
(A) A detailed assessment of the developer’s steps to comply with the requirements of this section.
(B) If applicable, any identified instances of noncompliance with the requirements of this section, and any recommendations for how the developer can improve its policies and processes for ensuring compliance with the requirements of this section.
(C) A detailed assessment of the developer’s internal controls, including its designation and empowerment of senior personnel responsible for ensuring compliance by the developer, its employees, and its contractors.
(D) The signature of the lead auditor certifying the results of the auditor.
. . .
(f)(2)(D) Beginning January 1, 2028, a certificate of compliance from an accredited third-party auditor the most recent audit report pursuant to subdivision (e).
- Clarify the Division’s role in auditing, specifying that it is to issue guidance on auditing best practices rather than manage an accreditation process. (Section 11547.6 (e)(2)(B).)
(B) Establish an accreditation process and relevant accreditation standards under which third-party auditors may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to their requirements under Section 22603 of the Business and Professions Code Issue guidance regarding best practices for conducting an audit pursuant to subdivision (e) of Section 22603 of the Business and Professions Code.
Additional amendments to the bill sought by the groups that would support the bill, if amended. The Electronic Frontier Foundation (EFF), which supports the bill, if amended, raises concerns about the abstract and confusing nature of the regulations for AI developers and the bill's focus on long-term catastrophic outcomes rather than immediate AI-enabled harms. They suggest that the bill should concentrate solely on the creation of CalCompute as a public cloud computing cluster. Additionally, EFF recommends ensuring that CalCompute is publicly owned and operated for public benefit, including serving California’s public universities. To prevent the diversion of taxpayer dollars to private entities, they propose explicit language to this effect. EFF also emphasizes the need for strong whistleblower protections and a private right of action to enhance enforcement.
Oakland Privacy, which also supports the bill, if it is amended, has expressed concerns regarding the definitions of AI safety incidents, which currently exclude those requested by users, potentially leaving unsafe incidents unaddressed. They also highlight the lack of independent third-party review for positive safety determinations (PSDs) and the limitation of enforcement to the State Attorney General, without provision for a private right of action. To address these issues, Oakland Privacy suggests removing the disclaimer in the definition of AI safety incidents, requiring independent third-party review of PSDs, and adding a limited private right of action for individuals harmed by critical safety incidents. Additionally, they recommend that the Frontier Model Division (FMD) provide or coordinate third-party expert witness testimony for civil suits.
The American Economic Liberties Project focuses on ensuring that CalCompute remains a truly public option, free from the influence of private donations or stakeholders. They express concerns about the limited enforcement capabilities due to scarce budgets and potential conflicts of interest and collusion within the Frontier Model Division. To foster public trust and enhance enforcement, they suggest clarifying CalCompute's public ownership and operation, adding a private right of action and whistleblower protections, and prohibiting conflicts of interest and interlocking directorates among regulated entities. Furthermore, they recommend strengthening "know your customer" (KYC) requirements for computing clusters and implementing ex ante structural separations between cloud services and model services to prevent market abuse.
Exemptions sought by various industries. The healthcare/life science industry suggests an exemption from the requirement to implement an emergency shutdown switch in AI systems. They argue that AI technologies in healthcare are deeply integrated into essential medical operations, such as diagnostics, treatment planning, and patient monitoring, and an emergency shutdown could result in severe disruptions to patient care. They propose tailored regulations that balance innovation with patient safety to avoid redundancy, increased compliance costs, and deterred innovation in developing new AI applications.
The research industry, particularly academic and non-profit research institutions, recommends an exemption from the stringent regulatory requirements of the bill. They argue that broad and prescriptive regulations, including the requirement for emergency shutdown capabilities, could stifle the open exchange of ideas, collaborative experimentation, and rapid development of innovative technologies. They suggest tailored regulations that recognize the unique context of academic research, taking into account existing ethical guidelines and review processes such as Institutional Review Boards (IRBs) and peer review, to support the advancement of AI while maintaining ethical standards (Stanford Center for AI Safety, Berkeley AI Research Lab).
Amendments sought by opponents. Focusing on high-risk uses of AI, rather than broadly regulating all AI technologies, is a suggestion made by multiple stakeholders to improve this measure. One primary recommendation is to concentrate regulatory efforts on specific applications of AI that have the potential to cause significant harm if misused, such as autonomous weapons, large-scale surveillance systems, and critical infrastructure controls. (BSA, The Software Alliance.)
Additionally, it has been suggested that the bill should encourage voluntary compliance with safety and security measures for lower-risk AI applications. The proponents of voluntary compliance argue that this approach would promote responsible AI development without imposing mandatory requirements that might be unnecessary for certain applications. Incentivizing voluntary adherence to best practices, according to Microsoft, can foster a culture of safety and ethics within the AI community. In summary, refining this bill to focus on high-risk AI uses involves targeting specific applications, developing a risk-based framework, aligning with industry best practices, and encouraging voluntary compliance for lower-risk applications. According to Abundance Institute, these suggestions aim to balance innovation with safety, ensuring that regulatory efforts are directed where they are most needed while supporting the broader development of AI technologies.
ARGUMENTS IN SUPPORT: Numerous advocacy groups and scholarly groups, including Civic AI Security Program, Higher Ground Labs, Lionheart Ventures, General Agents, Momentum, Panoplia Laboratories, Depict.AI, Mythos Ventures, Fathom Radiant, Elicit, FAR AI, Inc, Chapman University, Apart Research, KIRA Center for AI Risks & Impacts, Gladstone AI, General Proximity, Normative, Encultured AI, AI Safety Student Team (Harvard), Denizen, ML Alignment & Theory Scholars, ENH Alpha LLC, Causative Labs, Cambridge Boston Alignment Initiative, Redwood Research, New Media Studio, Paper Farms, Nonlinear, MIT AI Alignment, Lovable Labs Incorporated, White Space Marketing Group, AE Studio, and Safe AI Future, jointly write the following in support of this bill:
California has become a vibrant hub for artificial intelligence. Universities, startups, and technology companies are using Al to accelerate drug discovery, coordinate wildfire responses, optimize energy consumption, uncover rare minerals that produce clean energy, and enhance creativity. Artificial intelligence has enormous potential to benefit our state and the world. California must act now to ensure that it remains at the forefront of dynamic innovation in AI development.
At the same time, scientists, engineers, and business leaders at the cutting edge of this technology — including the three most cited AI researchers of all time — have repeatedly warned policymakers that failure to take appropriate precautions to prevent irresponsible AI development could have severe consequences for public safety and national security. California must ensure that the small handful of companies developing extremely powerful AI models —including companies explicitly aiming to develop “artificial general intelligence” — take reasonable care to prevent their models from causing very serious harms as they continue to produce models of greater and greater power.
Taken as a whole, SB 1047 strikes a good balance in addressing a variety of crucial areas in AI policy.
The Latino Community Foundation also writes in support:
SB 1047 introduces CalCompute, a publicly owned and operated cloud computing cluster aimed at fostering research and development for the betterment of society. Additionally, it mandates fair and transparent pricing and access to encourage market competition, all while establishing sensible guidelines for developers to ensure the safe and responsible creation of large-scale AI systems.
The technology industry is increasingly dominated by a handful of powerful corporations, resulting in significant financial gains and substantial influence. The rise of AI is exacerbating this trend, further consolidating power within the sector. Without equitable access to AI tools and resources, Latino-led nonprofit organizations, along with Latino small businesses and entrepreneurs, face the threat of being left behind in accessing vital public and private resources to expand their operations and effectively serve communities and engage consumers.
AI has the potential to greatly benefit California and its residents, from improving service delivery to communities to managing wildfires and saving energy. However, this potential remains unrealized if the current market dynamics continue to favor a select few corporations. California has the opportunity and responsibility to ensure that the economic prosperity stemming from the tech boom benefits everyone in the state.
ARGUMENTS IN OPPOSITION: Numerous stakeholders and leaders in the industry have written in opposition to SB 1047. California Life Sciences, Stanford Center for AI Safety, Berkeley AI Research Lab, Google LLC, California Chamber of Commerce, Abundance Institute, and Microsoft raise significant concerns that highlight common themes of concern, including stringent regulatory requirements, broad definitions, unfairly placed liability, and the potential stifling effect on open-source AI development.
One primary issue raised by groups in opposition is the unpredictability and uncontrollability of how AI technology might be used once it is deployed. Providers can implement comprehensive safeguards and ethical guidelines during the development phase, but they cannot fully anticipate or prevent every possible misuse by third parties. This unpredictability makes it unreasonable to hold developers accountable for actions beyond their control (Google LLC). Furthermore, such liability could stifle innovation and discourage the development of AI technologies. The fear of legal repercussions might lead AI developers to limit the release of new models or restrict access to their technologies, thereby hindering technological progress and the potential benefits of AI across various sectors. They argue that this could be particularly detrimental to startups and smaller companies, which may lack the resources to manage extensive liability risks (California Chamber of Commerce). Additionally, they assert that imposing liability on providers for third-party misuse creates a complex legal landscape. Determining the extent of a provider's responsibility in cases of misuse can be difficult, leading to lengthy and costly legal disputes. This complexity could deter investment in AI development and slow down the deployment of beneficial AI solutions in critical areas such as healthcare, education, and environmental protection (Abundance Institute). While it is crucial to ensure the safe and ethical use of AI technologies, placing liability on providers for third-party misuse raises significant issues of fairness, feasibility, and potential negative impacts on innovation and technological advancement (Microsoft).
Academic researchers write that they rely heavily on the open exchange of ideas, collaborative experimentation, and the rapid development of innovative technologies. Broad and prescriptive regulations, including the requirement for emergency shutdown capabilities, could stifle this collaborative environment and hinder scientific progress. Research institutions often operate with limited resources and budgets, and the additional compliance costs and administrative burdens could divert funds away from critical research activities. Tailored regulations recognizing the unique context of academic research, supported by existing ethical guidelines and review processes such as Institutional Review Boards (IRBs) and peer review, would better balance innovation with ethical considerations (Stanford Center for AI Safety, Berkeley AI Research Lab).
REGISTERED SUPPORT / OPPOSITION:
Support
Ae Studio
Ai Safety Student Team (HARVARD)
Apart Research
California State Council of Service Employees International Union (seiu California)
Cambridge Boston Alignment Initiative
Causative Labs
Center for Ai Safety Action Fund
Chapman University
Civic Ai Security Program
Consumer Watchdog
Denizen
Depict.ai
District Council of Iron Workers of The State of California and Vicinity
Economic Security Project Action
Elicit
Encode Justice
Encultured Ai
Enh Alpha LLC
Far Ai, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone Ai
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Loveable Labs Incorporated
Mit Ai Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panolia Laboratories
Panoplia Laboratories
Paper Farms
Redwood Research
Safe Ai Future
Techequity Collaborative
The Future Society
White Space Marketing Group
One individual
Support if Amended
American Economic Liberties Project
Electronic Frontier Foundation
Oakland Privacy
Opposition
8vdx
Abundance Institute
Acclamation Insurance Management Services
Adacta Labs
Aidy
Alec Action
Algolia
Allied Managed Care
Alloy Automation
American Consumer Institute
Andi
Andreessen Horowitz
Andy Ai
Anneal
Antifragile Research Dba Creatorml
Apriora
Argovox
Association of National Advertisers
Baseline Ai
Bilanc Finance INC.
Brighterway
California Chamber of Commerce
California Fuels and Convenience Alliance
California Land Title Association
California Manufacturers & Technology Association
California Manufacturers and Technology Association
Candoriq
Canonical Ai INC
Center for Data Innovation
Chamber of Progress
Chima
Circle Medical
Citron Labs
Civil Justice Association of California
Clarum
Clerky
Coalition of California Chambers – Orange County
Coalition of Small and Disabled Veteran Businesses
Cointracker
Competitive Enterprise Institute
Computer & Communications Industry Association
Computer and Communications Industry Association
Constructable
Consumer Technology Association
Context Fund
Curtsy
Cyble
Deconvolute Ai
Deferred
Dianahr
Digital First Project
Distill
Docsum
Double
Double Finance
Ecomback
Eden - Workplace Management Software
Edge
Ello
Elythea
Entori, INC
Envelope
Extensional, INC
Fetchflow
Figments Corp
Firebender Corp
Flasher Barricade Association
Flexdesk
Focal INC
Fondo
Gigaml
Gleam
Greptile
Happyrobot INC.
Hazel
Hedgy Labs
Insights Association
International Business Machines Corporation (IBM)
International Center for Law & Economics
Invert INC
Invoke Labs
Jo
K-scale Labs
Keywords Ai INC.
Kontractify, INC
Lightski
Linc Ai
Linum
Listening.com
Local Reg
Los Angeles Area Chamber of Commerce
Los Angeles Business Federation
Los Angeles County Business Federation (BIZ-FED)
Lovecast INC Dba Stably Ai
Lumona
Markprompt
Marr Labs Technologies, INC
Martian Mobility INC. Dba Telematica
Metriport
Miru
Movley
National Federation of Independent Business
National Federation of Independent Business (NFIB)
Octolane Ai
Oneshop
Onward
Openmart
Openpod
Orange Collective
Orange County Business Council
Pabio INC.
Patched Codes, INC
Patika Technology, INC
Patterns
Persist Ai
Phonely
Picnichealth
Playground
Poll Everywhere
Preloop INC.
Query Vary
R Street Institute
Rainforest Qa, INC.
Regology INC
Rejuvenation Technologies INC
Retell Ai
Revamp Ai
Reworkd Ai
Rinsed
Rippling
Risotto
Rocketlit INC.
Rootly
San Diego Regional Chamber of Commerce
San Juan Capistrano Chamber of Commerce
Sela
Sepal Ai
Sideguide Technologies INC (known As Firecrawl)
Silicon Valley Leadership Group
Simplify
Software & Information Industry Association
Software and Information Industry Association
Sorting Robotics
Southwest California Legislative Council
Spoken
Sprx
Squire Ai
Stack Ai
Sterling Road
Storyquest, INC.
Storyworth
Sway Finance
Sync Labs
Synova Life Sciences
Tailor
Taxgpt INC
Technet
Tensor
Tesorio INC
Tracecat
Trainy
Truewind Technologies, INC.
Turbolayer INC. Dba Automatica
Turboprop
Turing Labs
Tusk
Twine
Two Dots
Up Dog
Upkeep
Viva Labs
Walrus Tech INC
Willow
Wordware
Y Combinator
Zapier
Zaymo
Zep Ai
Oppose Unless Amended
Answerdotai, INC
California Life Sciences
Google
Q8 Empowering Change
Concerns
Meta
Analysis Prepared by: Alison Merrilees and Nicole Larson / JUD. / (916) 319-2334
Date of Hearing: June 18, 2024
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
SBPCA Bill Id:SB 1047
Author:(Wiener) – As Amended Ver:June 5, 2024
AS PROPOSED TO BE AMENDED
SENATE VOTE: 32-1
SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
SYNOPSIS
“We are current and former employees at frontier AI companies, and we believe in the potential of AI technology to deliver unprecedented benefits to humanity.
We also understand the serious risks posed by these technologies. These risks range from the further entrenchment of existing inequalities, to manipulation and misinformation, to the loss of control of autonomous AI systems potentially resulting in human extinction. AI companies themselves have acknowledged these risks, as have governments across the world and other AI experts.
We are hopeful that these risks can be adequately mitigated with sufficient guidance from the scientific community, policymakers, and the public. However, AI companies have strong financial incentives to avoid effective oversight, and we do not believe bespoke structures of corporate governance are sufficient to change this.”
The above paragraphs appear at the start of an open letter titled “A Right to Warn about Advanced Artificial Intelligence,” released on June 2, 2024 by a group of current and former OpenAI employees. The letter calls on AI companies to commit to various principles of openness and transparency, highlighting these companies’ “weak obligations” to share their knowledge of AI’s risks with the world.
This bill seeks to strengthen those obligations in order to mitigate the risk of catastrophic harms from AI models so advanced that they are not yet known to exist. SB 1047, as proposed to be amended, would require the developers of such models – which cost at least $100 million in computing power to train – to create good governance programs before initiating training. Following training, developers would be required to perform risk assessments on their models, subject to third party auditing, before using or releasing them. The bill creates a Division of Frontier Models in the Government Operations Agency to oversee this process. The bill also adds whistleblower protections; requires operators of computer clusters to implement “know your customer” requirements, including the ability to shut down any resources being used to train an advanced AI model; and creates a public computing cluster known as “CalCompute” in the Department of Technology. The Attorney General is charged with enforcing the bill’s requirements.
Proposed Committee amendments clarify and strengthen the bill’s provisions by, among other things: eliminating the limited duty exemption; adjusting the structure of the Frontier Model Division and placing it under the Board of Frontier Models; enabling the Frontier Model Division to update the definition of “covered model” beginning in 2027; requiring third party audits beginning in 2028; and clarifying the bill’s language and scope. The full text of the bill, as proposed to be amended, is included at the end of the analysis.
The net result is a strong framework for effective oversight in the face of “the loss of control of autonomous AI systems potentially resulting in human extinction.” Moving forward, the devil will be in the details. Certain issues require elaboration and refinement before the bill can be successfully implemented. The analysis details commitments by the author to continue working with the Committee on future amendments relating to enforcement provisions and whistleblower protections, positioning CalCompute within California’s university system, fleshing out the structure and responsibilities of the Board of Frontier Models, and further refinements to the bill’s terminology.
This bill has generated a great deal of commentary, consternation, and misconception. To set the record straight: SB 1047’s requirements are not onerous. For one, they only impact models that cost over $100 million in computing power to train – this threshold is baked into the definition of “covered model” provided by the bill and cannot be altered except by future legislative action. Secondly, the risk assessments developers would be required to perform broadly align with national and international guidelines, as well as procedures these developers already claim to implement. Finally, the bill only requires that developers implement shutdown capabilities for rogue models they themselves control. The open source community can rest easy knowing the models they download will not contain an immutable kill switch.
This bill does not create a state-sponsored licensing regime for AI; it does not ban the creation and use of AI above a certain compute threshold; and it does not create exorbitant costs for startups seeking to train large models. Performing basic risk assessments before using and releasing powerful, generally-capable models – and prohibiting their use only in extreme cases involving unreasonable risks of mass casualties or massive economic damages – is the bare minimum that Californians should expect of an industry claiming to have their best interests at heart.
This bill is sponsored by Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. It is supported by a variety of advocacy groups including the Future Society. It is opposed by a variety of industry trade associations including the California Chamber of Commerce, Technet, and the Chamber of Progress. If the bill passes this Committee, it will next be heard by the Assembly Judiciary Committee.
SUMMARY: Requires developers of especially advanced AI systems conduct risk assessments of those systems, to be overseen by the Frontier Model Division in the Government Operations Agency. Specifically, this bill:
- Defines “artificial intelligence” to mean an engineered or machine based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- Defines “fine-tuning” to mean adjusting the model weights of a trained model by exposing it to additional data.
- Before Jan. 1, 2027, defines “covered model” to mean either of the following:
- An artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations, the cost of which exceeds $100 million dollars when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
- An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than 3 times 10^25 integer or floating-point operations.
- On and after Jan. 1, 2027, defines “covered model” to mean either of the following:
- An artificial intelligence model trained using a quantity of computing determined by the Frontier Model Division, the cost of which exceeds $100 million dollars when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
- An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division.
- Defines “post-training modification” to mean modifying the capabilities of a covered model by any means, including, but not limited to, fine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
- Defines “covered model derivative” to mean any of the following:
- An unmodified copy of a covered model.
- A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
- Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding 3 times 10^25 integer or floating-point operations.
- On and after January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding a threshold determined by the Frontier Model Division.
- Defines “developer” to mean a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, or by fine-tuning an existing model using a sufficient quantity of computing power.
- Defines “critical harm” to mean any of the following:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
- Mass casualties or at least $500 million of damage resulting from cyberattacks on critical infrastructure, occurring either in a single incident or over multiple related incidents.
- Mass casualties, or at least $500 million of damage resulting from an artificial intelligence model autonomously engaging in conduct that would constitute a serious or violent felony under the Penal Code if were undertaken by a human with the requisite mental state.
- Other grave harms to public safety and security that are of comparable severity to the listed harms.
- Defines “full shutdown” to mean the cessation of operation of any of the following:
- The training of a covered model.
- A covered model.
- All covered model derivatives controlled by a developer.
- Requires that before a developer initially trains a covered model, they do all of the following:
- Implement protections to prevent unauthorized access, misuse, or unsafe post-training modification of the covered model and all associated covered model derivatives controlled by the developer.
- Implement the capability to enact a full shutdown.
- Implement a written safety and security protocol that does all of the following:
- Provides reasonable assurance that the developer will not produce a covered model or covered model derivative that poses an unreasonable risk of causing or enabling a critical harm.
- Identifies specific tests that would be sufficient to provide reasonable assurance that covered models and covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
- Describes the conditions under which a developer would enact a full shutdown.
- Describes the procedure by which the safety and security protocol may be modified.
- Implement the safety and security protocol and designate senior personnel to be responsible for ensuring compliance, including by conducting third party audits.
- Conduct an annual review of the safety and security protocol and make modifications as necessary.
- Provide a copy of the safety and security protocol to the Frontier Model Division when it is initially created, and within 10 business days of any modifications.
- Requires that before using a covered model or covered model derivative, or making a covered model or covered model derivative available for commercial or public use, the developer do all of the following:
- Assess whether the covered model is reasonably capable of causing a critical harm.
- Implement reasonable safeguards to prevent a covered model or associated covered model derivatives from causing a critical harm.
- Ensure, to the extent reasonably possible, that the actions of covered models and covered model derivatives can be attributed to them.
- Beginning Jan. 1, 2028, obtain a certificate of compliance from a third party auditor who has been accredited by the Frontier Model Division.
- Prohibits a developer from using a covered model or covered model derivative commercially or publicly, or making a covered model or covered model derivative available for commercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.
- Requires a developer of covered model to annually submit a certification of compliance to the Frontier Model Division for as long as the covered model or any covered model derivatives controlled by the developer are used commercially or publicly, or remain available for commercial or public use.
- Defines “artificial intelligence safety incident” to mean an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
- A covered model autonomously engaging in behavior other than at the request of a user.
- Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model.
- The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model.
- Unauthorized use of a covered model to cause or enable a critical harm.
- Requires a developer of a covered model to report each artificial intelligence safety incident to the Frontier Model Division within 72 hours of the developer learning of the safety incident.
- Defines “computing cluster” to mean a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
- Requires a person that operates a computing cluster to implement written policies and procedures to do all of the following when a customer utilizes compute resources sufficient to train a covered model:
- Obtain a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster.
- Assess whether the customer intends to train a covered model.
- Retain the customer’s internet protocol addresses used to access the cluster, along with the time and date of access or administrative action.
- Maintain the above records for 7 years and provide them to the Frontier Model Division or the Attorney General upon request.
- Implement the capability to enact a full shutdown of any resources being used to train a covered model.
- Requires a developer of a covered model that provides commercial access to it to provide a transparent, uniform, publicly available price schedule for the purchase of access to that model at a given level of quality and quantity subject to the developer’s terms of service and prohibits developers from engaging in unlawful discrimination or noncompetitive activity in determining price or access. Operators of computing clusters are required to do the same with respect to computing clusters. However, a person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
- Authorizes the Attorney General, if they have reasonable cause to believe that a person is in violation of these provisions, to bring an action seeking recovery of preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for the violation, including deletion of the covered model and the weights utilized in that model. Monetary damages to persons aggrieved and a court order for a full shutdown are also available. However, these remedies are only available in response to harm or an imminent risk or threat to public safety. The Attorney General may also recover a civil penalty in an amount not exceeding 10 percent of the cost, excluding labor, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor, to develop the covered model for any subsequent violation.
- Subjects liable defendants to joint and several liability and instructs the court to disregard corporate formalities under specific conditions:
- Where steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
- Where the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief.
- Prohibits a developer from preventing an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance. A developer shall not retaliate against an employee for disclosing such information. Developers must provide clear notice to all employees working on covered models of their rights and responsibilities under this section. The Attorney General may publicly release any complaint, or a summary of that complaint, if disclosure will serve the public interest.
- Clarifies that the duties and obligations imposed are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
- Establishes the Board of Frontier Models in the Government Operations Agency. Allows the Governor to appoint an executive director of the Board, subject to Senate confirmation, to exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged. Specifies that the board shall be composed of 5 members, as follows:
- A member of the open-source community, appointed by the Governor, subject to Senate confirmation.
- A member of the artificial intelligence industry, appointed by the Governor, subject to Senate confirmation.
- A member of academia, appointed by the Governor, subject to Senate confirmation.
- A member appointed by the Speaker of the Assembly.
- A member appointed by the Senate Rules Committee.
- Establishes the Frontier Model Division in the Government Operations Agency, under the direct supervision of the Board. Requires the Division to do the following:
- Annually review certifications received from developers.
- Advise the Attorney General on potential violations of the bill’s provisions.
- Establish an accreditation process for third party auditors.
- Publish anonymized safety reports.
- Issue guidance describing categories of AI safety events likely to constitute a state of emergency.
- Appoint and consult with an advisory committee for open-source AI, which shall:
- Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering the Frontier Model Division’s responsibilities.
- Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations related to developers of covered models.
- On or before Jan 1, 2027, and annually thereafter, issue regulations to update the definition of a “covered model” to ensure that it accurately reflects technological developments, scientific literature, and widely-accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of enabling critical harms. The updated definition shall contain the following:
- The initial compute threshold that an artificial intelligence model must exceed to be considered a covered model.
- The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.
- Tasks the Department of Technology with creating a public cloud computing cluster known as CalCompute through the commissioning of consultants with specified objectives, first of which is to study the safe and secure deployment of large-scale AI models. The consultants shall include representatives of national laboratories, public universities, and any relevant professional associations or private sector stakeholders. They shall evaluate and incorporate the following considerations into their plan:
- An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
- The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
- A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
- An analysis of the state’s investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
- A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
- Authorizes the Department of Technology to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate the establishment of CalCompute.
EXISTING LAW:
- Establishes the Government Operations Agency (Gov. Code § 12800.)
- Establishes the Department of Technology within the Government Operations Agency (Gov. Code § 12803.2.)
- Charges the Department of Technology with approving and overseeing information technology projects in the state (Gov. Code § 11546.)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS:
- AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.
In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.
GenAI tools can be released in open-source or closed-source formats by their creators. Open-source tools are publically available; researchers and developers can access their code and parameters. This accessibility increases transparency, but it has downsides: when a tool’s code and parameters can be easily accessed, they can be easily altered, and open-source tools have the potential to be used for nefarious purposes such as generating deepfake pornography and targeted propaganda. By comparison, closed-source tools are opaque with respect to their security features. It is harder for bad actors to generate illicit materials using these tools. But unlike open-source tools, closed-source tools are not subject to collective oversight because their inner workings cannot be examined by independent experts.
- Risk management. According to the Senate Judiciary Committee’s comprehensive analysis
of the bill:
With recent dramatic advances in the capabilities of AI systems, the need for frameworks for accountability and responsible development have become ever more urgent.
In January of 2017, AI researchers, economists, legal scholars, ethicists, and philosophers met in Asilomar, California to discuss principles for managing the responsible development of AI. The collaboration resulted in the Asilomar Principles. Aspirational rather than prescriptive, these 23 principles were intended to initiate and frame a dialogue by providing direction and guidance for policymakers, researchers, and developers. Its endorsers include 1,200 leading experts in the field of AI, including DeepMind founder Demis Hassabis and the late Stephen Hawking.[1]
The Legislature subsequently adopted ACR 215 (Kiley, Ch. 206, Stats. 2018), which added the State of California to that list by endorsing the Asilomar Principles as guiding values for the development of artificial intelligence and related public policy. In broad strokes, those principles aim to do the following:
- Research issues: create beneficial AI; direct funding toward beneficial innovation; maintain constructive and healthy exchanges between AI researchers and policymakers; promote a culture of trust, cooperation, and transparency among researchers and developers of AI; and avoid corner-cutting on safety standards.
- Ethics and values: promote safety, failure transparency, judicial transparency, and responsible innovation; align human values with innovation; protect privacy and liberty; ensure that the benefits and prosperity created by AI are broadly shared; maintain human control over AI; develop AI that supports rather than subverts social and civil processes; and avoid an AI arms race.
- Longer-term issues: avoid assumptions regarding the capabilities of AI; give AI its due attention; and recognize that its risks are potentially catastrophic or existential.
As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk
Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.
While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.[2]
More recently the Biden Administration has published its Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the design, use, and deployment of AI to protect the rights of the American public:
- Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
- Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
- Data Privacy: You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used. You should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used. Systems should not employ user experience and design decisions that obfuscate user choice or burden users with defaults that are privacy invasive. Consent should only be used to justify collection of data in cases where it can be appropriately and meaningfully given. Any consent requests should be brief, be understandable in plain language, and give you agency over data collection and the specific context of use; current hard-to-understand notice-and-choice practices for broad uses of data should be changed. Enhanced protections and restrictions for data and inferences related to sensitive domains, including health, work, education, criminal justice, and finance, and for data pertaining to youth should put you first. In sensitive domains, your data and related inferences should only be used for necessary functions, and you should be protected by ethical review and use prohibitions. You and your communities should be free from unchecked surveillance; surveillance technologies should be subject to heightened oversight that includes at least pre-deployment assessment of their potential harms and scope limits to protect privacy and civil liberties. Continuous surveillance and monitoring should not be used in education, work, housing, or in other contexts where the use of such surveillance technologies is likely to limit rights, opportunities, or access. Whenever possible, you should have access to reporting that confirms your data decisions have been respected and provides an assessment of the potential impact of surveillance technologies on your rights, opportunities, or access.
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.[3]
TechEquity, an organization committed to ensuring technology’s evolution benefits everyone equitably, has also laid out their straightforward AI Policy Principles:
- People who are impacted by AI must have agency to shape the technology that dictates their access to critical needs like employment, housing, and healthcare.
- The burden of proof must lie with developers, vendors, and deployers to demonstrate that their tools do not create harm—and regulators, as well as private [individuals], should be empowered to hold them accountable.
- Concentrated power and information asymmetries must be addressed in order to effectively regulate the technology.[4]
The need for thoughtful regulation and accountability is especially urgent with regard to the existential risks that many believe unfettered AI advancement poses. In response to these risks, the Future of Life Institute published an open letter early last year, calling for a pause on giant AI experiments:
Contemporary AI systems are now becoming human-competitive at general tasks, and we must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders. Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable. This confidence must be well justified and increase with the magnitude of a system's potential effects. OpenAI's recent statement regarding artificial general intelligence, states that “At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models.” We agree. That point is now.
Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.[5]
Signatories to the letter include Stuart Russell, Berkeley, Professor of Computer Science, director of the Center for Intelligent Systems, and co-author of the standard textbook “Artificial Intelligence: a Modern Approach”; Elon Musk, CEO of SpaceX, Tesla & X; and Steve Wozniak, Co-founder, Apple.
Subsequent to that letter, the Center for AI Safety released another open letter signed by a wide-ranging group of industry leaders, researchers, and engineers working in AI that highlighted the existential risk posed by unethical AI development and the urgency of the issue. The statement simply read: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”[6]
This was signed by the most cited researchers of AI, including Dr. Yoshua Bengio and Dr. Geoffrey Hinton; both Turing Award winners and considered the “godfathers” of modern AI. In addition, prominent executives at the leading AI development companies also signed on, including Ilya Sutskever, co-founder and chief scientist, OpenAI; Sam Altman, chief executive of OpenAI; Demis Hassabis, chief executive of Google DeepMind; and Dario Amodei, chief executive of Anthropic.
While the future is unclear, the need to respond to these potential harms now is evident. The Center for New American Security puts a fine point on it:
While there is significant uncertainty in how the future of AI develops, current trends point to a future of vastly more powerful AI systems than today’s state of the art. The most advanced systems at AI’s frontier will be limited initially to a small number of actors but may rapidly proliferate. Policymakers should begin to put in place today a regulatory framework to prepare for this future. Building an anticipatory regulatory framework is essential because of the disconnect in speeds between AI progress and the policymaking process, the difficulty in predicting the capabilities of new AI systems for specific tasks, and the speed with which AI models proliferate today, absent regulation. Waiting to regulate frontier AI systems until concrete harms materialize will almost certainly result in regulation being too late.
- What this bill, as proposed to be amended, would do. SB 1047 would require developers of “covered models” to conduct risk assessments on those models before using them or making them available for public use. Until 2027, models are considered “covered models” if they are trained using roughly ten times as much computing power as any model that exists today and the costs of that quantity of compute power exceeds $100 million.
Beginning in 2027, the Frontier Model Division may adjust the compute threshold determining which models are considered “covered models.” Developers are prohibited from using or releasing covered models that pose an unreasonable risk of causing or enabling “critical harms,” resulting in mass casualties or $500 million in damages, as specified. Beginning January 1, 2028, developers of covered models will be subject to third party audits.
The bill requires the operators of computing clusters to know their customers and implement the ability to shut down any hardware used to train a covered model.
The bill creates the Frontier Model Division in the Government Operations Agency, under control of the Frontier Model Board. The Division is charged with various duties related to overseeing the development of advanced AI in the state, including setting thresholds for the definition of “covered model” and creating a third party auditor accreditation program.
The bill creates CalCompute, a public computing cluster located in the Department of Technology.
Finally, the bill also adds whistleblower protections related to covered models and gives the Attorney General civil enforcement authority.
- Author’s statement.
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a public AI research cluster that will allow startups, researchers, and community groups to participate in the development of large-scale AI systems. By providing a broad range of stakeholders with access to the AI development process, CalCompute will help align large-scale AI systems with the values and needs of California communities.
- Analysis. Rather than retreading ground previously covered by Senate Judiciary Committee, this analysis focuses on outlining and explaining the Committee’s proposed amendments.
Updating the definition of “covered model.” In order to be considered a “covered model” under SB 1047, a model must meet one of two conditions:
- The model is a new model that is initially trained using a quantity of computing power exceeding 10^26 integer or floating-point operations (and costing at least $100 million).
- The model is an existing model that is fine-tuned (it receives additional training) using a quantity of computing power exceeding 3 times 10^25 integer or floating-point operations.
The most advanced contemporary models are trained using roughly a tenth of the compute required to be considered a “covered model” under this bill.[7] Could ChatGPT4 be capable of causing a “critical harm”? Possible, but unlikely. However, as the training data that inform AI become more useful, and the architectures that underpin these systems become more carefully constructed, it is not unthinkable that a 10^25 model will one day be capable of causing such a harm. If this comes to pass, the definition of “covered model” will need to be updated. The European Union, importantly, sets a comparable threshold at 10^25 integer or floating-point operations.[8]
Alternatively, 10^26 models may end up being completely harmless. Five years from now it may make no sense for models of this size to be rigorously tested while 10^27, 10^28, and 10^29 models exist in the ecosystem. The definition of “covered model” would benefit from receiving an update in this case as well, if for no other reason than to avoid overwhelming the Frontier Model Division.
The fine-tuning threshold outlined in condition 2 has a more serious issue: fine-tuning takes far less compute than initially training a model. Condition 2 requires that a model receive 3 times 10^25 integer or floating-point operations worth of training before becoming a “new” covered model. Upon meeting this threshold, responsibility for the model switches over to the party that fine-tuned the model. This means the original developer is responsible for guaranteeing the safety of a covered model within a fine-tuning threshold of nearly 1/3 the original compute that went into its training.
It is not clear that this is possible. A set of leaked specifications for ChatGPT4 (which may not be entirely reliable, but are likely in the right ballpark) suggest that ChatGPT4 was initially trained using 13 trillion tokens.[9] OpenAI claims that their models’ behavior begins to change when fine-tuned using 50-100 training samples, or roughly 200,000 tokens (if each sample is ~2048 tokens).[10] There is a clear disconnect here. As it becomes more apparent how much initial compute is required for a model to acquire harmful capabilities, it may also become apparent how much fine-tuning compute allows a safe model to become unsafe.
Committee amendments would create a mechanism to update these thresholds in light of new evidence, through regulations passed by the Frontier Model Division beginning in 2027. Importantly, the Division’s discretion in adjusting the thresholds is considerably constrained. In updating the threshold, through regulation, the Division must ensure the update accurately reflects technological developments, scientific literature, and widely-accepted national and international standards, and that it applies to AI that pose the greatest risk of enabling critical harms. Additionally, the Division would not have the ability to alter the “cost” threshold of training a covered model. If the compute cost of training a 10^26 model drops below $100 million in future, and 10^26 models are demonstrated to be broadly capable of causing critical harms, this cost threshold may need to be updated through legislative action.
Eliminating the “limited duty exemption.” The bill in print contains a mechanism for developers to self-certify that their models possess no harmful capabilities, called the “limited duty exemption.” If a model qualifies for one of these “exemptions,” it is not subject to any of downstream requirements of the bill. Confusingly, developers are asked to make this assessment before a model has been trained—that is, before it exists. Writing in opposition, the California Chamber of Commerce explains why this puts developers in an impossible position:
SB 1047 still makes it impossible for developers to actually determine if they can provide reasonable assurance that a covered model does not have hazardous capabilities and therefore qualifies for limited duty exemption because it requires developers to make the determination before they initiate training of the covered model . . . Because a developer needs to test the model by training it in a controlled environment to make determination that a model qualifies for the exemption, and yet cannot train a model until such a determination is made, SB 1047 effectively places developers in a perpetual catch-22 and illogically prevents them from training frontier models altogether.
Furthermore, SB 1047 is predicated on the notion that advanced AI models are inherently risky above a certain compute threshold. It makes little sense to fully exempt a subset of these models from the provisions of this bill. Committee amendments abolish the limited duty exemption. Under this new framework, all developers that train covered models must create governance programs and perform basic risk assessments.
Clarifying the definition of “developer.” The version of the bill in print defines “developer” to mean a person that creates, owns, or otherwise has responsibility for an artificial intelligence model. It then excludes a number of individuals from this definition: “a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.” This creates various loopholes: if OpenAI purchases a computing cluster and exclusively uses that cluster to train ChatGPT5, are they then not considered a developer under this bill? Committee amendments simplify the definition of developer, focusing on persons who are in the position to comply with the bill’s requirements.
Clarifying the scope of a “full shutdown.” SB 1047’s “full shutdown” requirement has been a source of constant consternation for the open-source community. CalChamber explains:
Under SB 1047, developers must build “full shutdown” capabilities into their models and may be held liable for downstream uses over which they have no control, impeding their ability to open-source their models. Ultimately, liability should rest with the user who intended to do harm, as opposed to automatically defaulting to the developer who could not foresee, let alone block, any and all conceivable uses of a model that might do harm. While recent amendments seemingly seek to narrow what is meant by “full shutdown” capabilities, the exclusions are unnecessarily difficult to interpret as drafted (full shutdown “does not mean the cessation of operation of a covered model to which access was granted pursuant to a license that was not created by the licensor…”) and altogether insufficient.
Committee amendments simplify and clarify the definition of “full shutdown” such that the shutdown capability can be implemented into hardware used to train or run a model, rather than the model itself. The amendments also serve to exclude covered model derivatives that are outside of the developer’s control.
Reframing the bill in terms of “risk of causing or enabling a critical harm.” SB 1047 previously alternated between the terms “hazardous capabilities” and “critical harms” when describing the dangers associated with advanced AI. The definition of “critical harm” simply pointed to the definition of “hazardous capability,” and the definition of hazardous capability was effectively “the capability of a covered model to be used to enable one of [a list of harms].” Committee amendments simplify the bill’s language by couching everything in terms of a covered model’s capability to cause or enable various critical harms.
Adjusting a “critical harm” exemption. The version of the bill in print bakes the following exemption into the definition of “critical harm/hazardous capability”: hazardous capability means “the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model.” This exemption is crafted broadly and leads to unintended consequences. For example: an autonomous robot that goes on a stabbing spree might not be covered by this bill, as the creator of the model could just as easily have picked up a knife. According to the author and sponsors, the intent of this exemption is to cover information that a model could provide, but that could easily be found elsewhere (such as online.) Committee amendments narrow the scope of this exemption by focusing on harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible. This solution is imperfect: for example, it results in advanced AI effectively inheriting various protections afforded to the internet under Section 230. It also fails to account for situations where information found online originated from another advanced AI. The author has committed to continue to work with the Committee on refining this language.
Closing a “derivative model” loophole. The bill in print contains a loophole related to the definition of “derivative model.” A derivative model includes “a modified or unmodified copy of an artificial intelligence model,” and derivative models are exempted from most of the bill’s requirements. However, the definition of derivative model does not specify that this model must be owned or operated by an entity other than the original developer. This means that a developer could train a model, copy/paste it to a different set of hardware (but otherwise leave it unaltered), and be considered exempt from the provisions of the bill due to the copy being a “derivative.” Committee amendments close this loophole by recasting the bill in terms of “covered models and covered model derivatives” and requiring developers to be responsible for both.
Clarifying the bill’s prohibition on the use of models with unreasonable risks of causing critical harms. The bill in print requires the following:
Before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, a developer of the nonderivative version of the covered model shall do all of the following . . . Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm. (Emphasis added.)
This provision would appear to ban the deployment of dangerous covered models. However, it is not entirely clear due to the bill’s use of the term “refrain,” which the Merriam-Webster dictionary defines as “keep[ing] oneself from doing, feeling, or indulging in something and especially from following a passing impulse.” In other words, the bill in print arguably does not ban the deployment of a model – it only requires that a developer try their best not to release it. Committee amendments clarify that a developer is prohibited from using or releasing covered models “if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.”
Requiring third-party auditing. The bill in print directs the Frontier Model Division to create an “optional” accreditation process for third party auditing. Committee amends retain this function of the Frontier Model Division, but make third party auditing mandatory. Beginning January 1, 2028, developers must obtain a certificate of compliance from an accredited third party auditor before they can use or make available covered models or covered model derivatives they have trained.
Adjusting the structure and placement of the Frontier Model Division. Committee amendments give the Frontier Model Division two new responsibilities under this bill: accrediting third party auditors, and adjusting the compute thresholds in the definition of “covered model.” These changes empower the Division to both determine which entities are subject to the provisions of this bill, and determine how strictly to enforce compliance. To ensure that this process is done with proper transparency, accountability, and public participation, Committee amendments create a “Board of Frontier Models” in the Government Operations Agency to supervise and direct the Division. Five members sit on this board: three Governor appointees, one Senate appointee, and one Assembly appointee. Of the Governor’s appointees, all three must receive Senate confirmation. One must originate from academia, one from industry, and one from the open source community. By building various perspectives into the leadership of the Board, this Committee hopes to ensure the Division’s work remains fair and free of bias. Committee amendments also move the Division itself from the Department of Technology to the Government Operations Agency.
Various language changes. Committee amendments update language throughout SB 1047 to clarify the bill’s scope and requirements.
Author commitments. In addition to the changes outlined above, the author has agreed to continue working with the Committee on several pieces of SB 1047. These are briefly outlined below:
- Restructuring the enforcement scheme in collaboration with the Assembly Judiciary Committee.
- Expanding whistleblower protections in collaboration with the Assembly Judiciary Committee.
- Positioning CalCompute in California’s university system, rather than in the Department of Technology.
- Fleshing out the structure and responsibilities of the Board of Frontier Models.
- Workshopping a narrow exemption for “critical harms” that relates to the ability of these models to produce information.
- Adjusting language to clarify the bill’s provisions and bring various terminology in line with industry standards, without affecting the overall impact of the bill.
- Full text as proposed to be amended.
CHAPTER 22.6. Safe and Secure Innovation for Frontier Artificial Intelligence Models
22602. As used in this chapter:
(a) “Advanced persistent threat” means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, including, but not limited to, cyber, physical, and deception, to generate opportunities to achieve its objectives that are typically to establish and extend its presence within the information technology infrastructure of organizations for purposes of exfiltrating information or to undermine or impede critical aspects of a mission, program, or organization or place itself in a position to do so in the future.
(b) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infers,infer from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(c) “Artificial intelligence safety incident” means an incident that demonstrably increases the risk of a critical harm occurring by means of any of the following:
(1) A covered model autonomously engaging in behavior other than at the request of a user that materially increases the risk of a hazardous capability being used.
(2) Theft, misappropriation, malicious use, inadvertent release, unauthorized access, or escape of the model weights of a covered model that is not the subject of a limited duty exemption.
(3) The critical failure of technical or administrative controls, including controls limiting the ability to modify a covered model that is not the subject of a limited duty exemption.
(4) Unauthorized use of the hazardous capability of a covered model to cause or enable a critical harm.
(d) “Computing cluster” means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
(e) “Covered guidancemodel” means either of the following:
(1) Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(2) Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
(f) (1) “Covered model” means an(1) Before January 1, 2027, “covered model” means either of the following:
(A) An artificial intelligence model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations, and the cost of that quantity of computing power would exceedwhich exceeds one hundred million dollars ($100,000,000) ifwhen calculated using the average market pricesprice of cloud compute at the start of training as reasonably assessed by the developer at the time of training.
(2(B) An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power equal to or greater than 3 times 10^25 integer or floating-point operations.
(2) Except as provided in subparagraph (C), on and after January 1, 2027, “covered model” means any of the following:
(A) An artificial intelligence model trained using a quantity of computing power determined by the Frontier Model Division pursuant to Section 11547.6 of the Government Code, the cost of which exceeds one hundred million dollars ($100,000,000) when calculated using the average market price of cloud compute at the start of training as reasonably assessed by the developer.
(B) An artificial intelligence model created by fine-tuning a covered model using a quantity of computing power that exceeds a threshold determined by the Frontier Model Division.
(C) If the Frontier Model Division does not adopt a regulation governing subparagraphs (A) and (B) by January 1, 2027, the definition of “covered model” in paragraph (1) continues to be in effect until the regulation is adopted.
(3) On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(g) “Critical harm” means a harm listed in paragraph (1) of subdivision (n).
(i) (1) “Derivative(f) “Covered model” means an artificial intelligence model that is a derivative of another artificial intelligence model, including either “ means any of the following:
(A) A modified or1) An unmodified copy of an artificial intelligencea covered model.
(2) A copy of a covered model that has been subjected to post-training modifications unrelated to fine-tuning.
(3) (A) Before January 1, 2027, a copy of a covered model that has been fine-tuned using a quantity of computing power not exceeding 3 times 10^25 integer or floating-point operations.
(B) A combinationOn and after January 1, 2027, a copy of an artificial intelligencea covered model with other software.
(2) “Derivative model” does not include either of the following:
(A) An entirely independently trained artificial intelligence model.
(B) An artificial intelligence model, including one combined with other software, that ishas been fine-tuned using a quantity of computing power greater than 25 percent of the quantity of computing power, measured in integer or floating-point operations, used to train the original modelnot exceeding a threshold determined by the Frontier Model Division.
(j) (1) “Developer” means a person that creates, owns, or otherwise has responsibility for an artificial intelligence model.
(2) “Developer” does not include a third-party machine-learning operations platform, an artificial intelligence infrastructure platform, a computing cluster, an application developer using sourced models, or an end-user of an artificial intelligence model.
(k) “Fine tuning” means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
(l) “Frontier Model Division” means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(m) (1) “Full shutdown” means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within the custody, control, or possession of a nonderivative model developer or a person that operates a computing cluster, including any computer or storage device remotely provided by agreement.
(2) “Full shutdown” does not mean the cessation of operation of a covered model to which access was granted pursuant to a license that was not granted by the licensor on a discretionary basis and was not subject to separate negotiation between the parties.
(n) (1) “Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model that does not qualify for a limited duty exemption:
(C) If the Frontier Model Division does not adopt a regulation governing subparagraph (B) by January 1, 2027, the quantity of computing power specified in subparagraph (A) shall continue to apply until the regulation is adopted.
(4) A copy of a covered model that has been combined with other software.
(g) (1) “Critical harm” means any of the following harms caused or enabled by a covered model or covered model derivative:
(A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
(B) AtMass casualties or at least five hundred million dollars ($500,000,000) of damage throughresulting from cyberattacks on critical infrastructure via , occurring either in a single incident or over multiple related incidents.
(C) AtMass casualties or at least five hundred million dollars ($500,000,000) of damage byresulting from an artificial intelligence model that autonomously engagesengaging in conduct that would violateconstitute a serious or violent felony under the Penal Code if undertaken by a human with the necessaryrequisite mental state and causes either of the following:.
(i) Bodily harm to another human.
(ii) The theft of, or harm to, property.
(D) Other grave threatsharms to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.
(2) “Hazardous capability” includes a capability described in paragraph (1) even if the hazardous capability would not manifest but for fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
(2) “Critical harm” does not include harms caused or enabled by information that a covered model outputs if the information is otherwise publicly accessible.
(3) On and after January 1, 2026, the dollar amounts in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.
(h) “Critical infrastructure” means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
(o) “Limited duty exemption” means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model, which applies if a developer can provide reasonable assurance that a covered model does not have a hazardous capability and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
(p) “Machine-learning operations platform” means a solution that includes a combined offering of necessary machine-learning development capabilities, including exploratory data analysis, data preparation, model training and tuning, model review and governance, model inference and serving, model deployment and monitoring, and automated model retraining.
(q(i) “Developer” means a person that performs the initial training of a covered model either by training a model using a sufficient quantity of computing power, or by fine-tuning an existing covered model using sufficient quantity of computing power pursuant to subdivision (e).
(j) “Fine-tuning” means adjusting the model weights of a trained covered model by exposing it to additional data.
(k) “Frontier Model Division” means the Frontier Model Division created pursuant to Section 11547.6 of the Government Code.
(l) “Full shutdown” means the cessation of operation of any of the following:
(1) The training of a covered model.
(2) A covered model.
(3) All covered model derivatives controlled by a developer.
(m) “Model weight” means a numerical parameter established through training in an artificial intelligence model that is adjusted through training and that helps determine how input information impacts a model’s outputinputs are transformed into outputs.
(rn) “Open-source artificial intelligence model” means an artificial intelligence model that is made freely available and that may be freely modified and redistributed.
(so) “Person” means an individual, proprietorship, firm, partnership, joint venture, syndicate, business trust, company, corporation, limited liability company, association, committee, or any other nongovernmental organization or group of persons acting in concert.
(t) “Posttraining p) “Post-training modification” means the modification ofmodifying the capabilities of an artificial intelligencea covered model after the completion of training by any means, including, but not limited to, initiating additional trainingfine-tuning, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
(uq) “Reasonable assurance” does not mean full certainty or practical certainty.
(vr) “Safety and security protocol” means documented technical and organizational protocols that meet both of the following criteria:
(1) The protocols are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative modelscovered model derivatives.
(2) The protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developer’s covered model.
22603.
(a) Before initiating training ofa developer initially trains a covered model that is not a derivative model, a developer of that covered model may determine whether, the covered model qualifies for a limited duty exemption.
(1) In making the determination authorized by this subdivision, a developer shall incorporate all applicable covered guidance.
(2) A developer may determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant under subdivision (f) of Section 22602 and has an equal or lesser general capability than either of the following:
(A) A noncovered model that manifestly lacks hazardous capabilities.
(B) Another model that is the subject of a limited duty exemption.
(3) Upon determining that a covered model qualifies for a limited duty exemption, the developer of the covered model shall submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination.
(4) A developer that makes a good faith error regarding a limited duty exemption shall be deemed to be in compliance with this subdivision if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the artificial intelligence model until the developer is otherwise in compliance with subdivision (b).
(b) Before initiating training of a covered model that is not a derivative model and is not the subject of a limited duty exemption, and until that covered model is the subject of a limited duty exemption, the developer of that covered model shall do all of the following:
(1) Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse of, or unsafe modificationpost-training modifications of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the and all covered model weights fromderivatives controlled by the developer’s custody,developer that are appropriate in light of the risks associated with the covered model, including from advanced persistent threats or other sophisticated actors.
(2) Implement the capability to promptly enact a full shutdown of the covered model.
(3) Implement all covered guidance.
(4) Implement a written and separate safety and security protocol that does all of the following:
(A) Provides reasonable assurance that ifIf a developer complies with itsthe safety and security protocol, either of the following will apply:
(i) Theprovides reasonable assurance that the developer will not produce a covered model with a hazardous capability or enable the production of a or covered model derivative model with a hazardous capability.
(ii) The safeguards enumerated in the protocol will be sufficient to preventthat poses an unreasonable risk of causing or enabling a critical harms from the exercise of a hazardous capability in a covered model.harm.
(B) States compliance requirements in an objective manner and with sufficient detail and specificity to allow the developer or a third party to readily ascertain whether the requirements of the safety and security protocol have been followed.
(C) Identifies specific tests and test results that would be sufficient to provide reasonable assurance that a covered model does not have a hazardous capability and will not come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications, and in addition does all of the following:of the following:
(i(i) That a covered model does not pose an unreasonable risk of causing or enabling a critical harm.
(ii) That covered model derivatives do not pose an unreasonable risk of causing or enabling a critical harm.
(D) Describes in detail how the testing procedure incorporates fine tuning and posttrainingassesses the risks associated with post-training modifications performed by third-party experts intending to demonstrate those abilities.
(ii) Describes in detail how the testing procedure incorporates the possibility of posttraining modifications.
(iii) Describes in detail how the testing procedure incorporates the requirement for reasonable margin for safety.
(iv(E) Describes in detail how the testing procedure addresses the possibility that a covered model can be used to make posttrainingpost-training modifications or create another covered model in a manner that may generate hazardous capabilities.
(vF) Provides sufficient detail for third parties to replicate the testing procedure.
(DG) Describes in detail how the developer will meet requirements listedfulfill their obligations under paragraphs (1), (2), (3), and (5).this chapter.
(E) If applicable, describes(H) Describes in detail how the developer intends to implement the safeguards and requirements referenced in paragraph (1) of subdivision (d).
(FI) Describes in detail the conditions thatunder which a developer would require the execution ofenact a full shutdown.
(GJ) Describes in detail the procedure by which the safety and security protocol may be modified.
(H) Meets other criteria stated by the Frontier Model Division in guidance to achieve the purpose of maintaining the safety of a covered model with a hazardous capability.
(5(4) Ensure that the safety and security protocol is implemented as written, including, at a minimum, by designating senior personnel to be responsible for ensuring implementationcompliance by employees and contractors working on a covered model, monitoring and reporting on implementation, and conducting audits, including through third parties as appropriateparty auditors.
(65) Provide a copy of the safety and security protocol to the Frontier Model Division.
(76) Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
(87) If the safety and security protocol is modified, provide an updated copy to the Frontier Model Division within 10 business days.
(9) Refrain8) Implement other reasonable measures to prevent covered models and covered model derivatives from initiating training posing unreasonable risks of causing or enabling critical harms.
(b) Before using a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the or covered model, or a derivative, or making a covered model based on it, to cause a critical harm.
(10) Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(c) (1) Upon completion of the training of a or covered model that is not the subject of a limited duty exemption under subdivision (a) and is not a derivative modelavailable for commercial or public use, the developer shall perform capability testing sufficient to determine if a limited duty exemption applies with respect to the covered model pursuant to its safety and security protocol.
(2) Upon determining if a limited duty exemption applies with respect to the covered model, a developer of the covered model shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes both of the following:
(A) The basis for the developer’s determination whether a limited duty exemption applies.
(B) The specific methodology and results of the capability testing undertaken pursuant to this subdivision.
(d) Before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, a developer of the nonderivative version of theof a covered model shall do all of the following:
(1) (1) Assess whether the covered model is reasonably capable of causing a critical harm.
(2) Implement reasonable safeguards and requirements, informed byto prevent the training and testing process, to do all of the following:
(A) Prevent an individual from being able to use the hazardous capabilities of thecovered model, or a derivative and covered model, to cause derivatives from causing a critical harm.
(B) Prevent an individual from being able to use the model to create a derivative model that is used to cause a critical harm.
(C(3) Ensure, to the extent reasonably possible, that the covered model’s actions and the actions of covered model derivatives, as well as any resulting critical harms resulting from their actions, can be accurately and reliably attributed to it and any user responsiblethem.
(4) Beginning January 1, 2028, obtain a certificate of compliance from a third party auditor who has been accredited pursuant to 11547.6 of the Government Code.
(c) A developer shall not use a covered model commercially or publicly, or make a covered model or a covered model derivative available for those actionscommercial or public use, if there is an unreasonable risk that the covered model or covered model derivative can cause or enable a critical harm.
(2) (d) A) Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
(B) If a developer provides access to the derivative model in a form that makes fine tuning possible, provide information to developers of that derivative model in a manner that will enable them to determine whether they have done a sufficient amount of fine tuning to meet the threshold described in subparagraph (B) of paragraph (2) of subdivision (i) of Section 22602.
(3) Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
(4) Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
(e) A developer of a nonderivative covered model shall periodicallyannually reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
(fe) (1) A developer of a nonderivative covered model that is not the subject of a limited duty exemption shall annually submit to the Frontier Model Division an annuala certification under penalty of perjury of compliance with the requirements of this section signed by the chief technology officer, or a more senior corporate officer, in a format and on a date as prescribed by the Frontier Model Division. This paragraph applies as long as the covered model or any covered model derivatives controlled by the developer remain in commercial or public use, or remain available for commercial or public use.
(2) In a certification submitted pursuant to paragraph (1), a developer shall specify or provide, at a minimum, all of the following:
(A) The nature and magnitude of hazardous capabilitiescritical harms that the covered model possesses or covered model derivatives may reasonably possesscause or enable, and the outcome of capability testingthe assessment required by subdivision (cb).
(B) An assessment of the risk that compliance with the safety and security protocol may be insufficient to prevent harms from the exercise of the covered model’s hazardous capabilitiesmodel or covered model derivatives from causing critical harms.
(C) Other information useful to accomplishing the purposes of this subdivision, as determined by the Frontier Model Division.
(g(C) A description of the process used by the signing officer to verify compliance with the requirements of this section, including a description of the materials reviewed by the signing officer, a description of testing or other evaluation performed to support the certification, and the contact information of any third parties relied upon to validate compliance.
(D) Beginning January 1, 2028, a certificate of compliance from an accredited third party auditor.
(f) (1) A developer of a nonderivative covered model shall report each artificial intelligence safety incident affecting thatthe covered model and any derivative version of thator any covered model within the custody, control, or possession of the derivatives controlled by the developer, as described in subdivision (m) of Section 22602, to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
(2) The report required by this subdivision shall be made not later thanwithin 72 hours afterof the developer learns that an learning of the artificial intelligence safety incident has occurred, or within 72 hours of the developer learnslearning facts sufficient to establish a reasonable belief that an artificial intelligence safety incident has occurred.
(h) (1) (A) Reliance on an unreasonable limited duty exemption does not relieve a developer of its obligations under this section.
(B) A determination that a covered model qualifies for a limited duty exemption that results from a good faith error reported pursuant to paragraph (4) of subdivision (a) is not an unreasonable limited duty exemption.
(2) A limited duty exemption is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination.
(3) A risk of harm or weakness in capability testing is reasonably foreseeable, if, by the time that(g) A developer shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance with the requirements of this section no more than 30 days after making a covered model or covered model derivative available for commercial or public use for the first time. A developer need not submit a certification for a covered model derivative if the developer has already submitted a certification for the applicable covered model.
(h) In fulfilling their obligations under this chapter, a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by either of the following:
(A) Any other developer of a comparable or comparably powerful model through risk assessment, capability testing, or other means.
(B) By theshall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, the Frontier Model Division, or any independentand other reputable standard-setting organization or capability-testing organization cited by either of those entities.organizations.
22604. (a) A person that operates a computing cluster shall implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
(a1) Obtain a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster, including all of the following:
(1A) The identity of that prospective customer.
(2B) The means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier.
(3C) The email address and telephonic contact information used to verify a prospective customer’s identity.
(b2) Assess whether a prospective customer intends to utilize the computing cluster to deploytrain a covered model.
(c) Annually(3) If a customer repeatedly utilizes computer resources that would be sufficient to train a covered model, validate the information initially collected pursuant to subdivision (a) and conduct the assessment required pursuant to subdivision (b).) prior to each utilization.
(d(4) Retain a customer’s Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
(5) Maintain for seven years and provide to the Frontier Model Division or the Attorney General, upon request, appropriate records of actions taken under this section, including policies and procedures put into effect.
(e6) Implement the capability to promptly enact a full shutdown in the event of an emergency.
(f) Retain a of any resources being used to train or operate such customer’s Internet Protocol addresses used for access or administration and the date and time of each access or administrative actionadministered models.
(b) A person that operates a computing cluster shall consider applicable guidance from the Frontier Model Division, National Institute of Standards and Technology, and other reputable standard-setting organizations.
[…]
SEC. 4. Section 11547.6 is added to the Government Code, to read:
11547.6. (a) As used in this section:
(1) “Hazardous capabilityCritical harm” has the same meaning as defined in Section 22602 of the Business and Professions Code.
(2) “Limited duty exemption” has the same meaning as defined in Section 22602 of the Business and Professions Code.
(b(b) There is hereby established the Board of Frontier Models. The board shall be housed in the Government Operations Agency and shall independent of the Department of Technology. The Governor may appoint an executive officer of the board, subject to Senate confirmation, who shall hold the office at the pleasure of the Governor. The executive officer shall be the administrative head of the board and shall exercise all duties and functions necessary to ensure that the responsibilities of the board are successfully discharged.
(c) Commencing January 1, 2026, the Board shall be composed of 5 members, as follows:
(1) A member of the open-source community, appointed by the Governor, subject to Senate confirmation.
(2) A member of the artificial intelligence industry, appointed by the Governor, subject to Senate confirmation.
(3) A member of academia, appointed by the Governor, subject to Senate confirmation.
(4) A member appointed by the Speaker of the Assembly.
(5) A member appointed by the Senate Rules Committee.
(d) The Frontier Model Division is hereby created within the Department of TechnologyGovernment Operations Agency under the direct supervision of the Board.
(ce) The Frontier Model Division shall do all of the following:
(1) Annually review certification reports received from developers pursuant to Section 22603 of the Business and Professions Code and publicly release summarized findings based on those reports.
(2) Advise the Attorney General on potential violations of this section or Chapter 22.6 (commencing with Section 22602) of Division 8 of the Business and Professions Code.
(3) (A) Issue guidance, standards, and best practices necessary to prevent unreasonable risks fromof covered models with hazardous capabilitiesand covered model derivatives causing critical harms, including, but not limited to, more specific components of or requirements under the duties required under Section 22603 of the Business and Professions Code.
(B) Establish an optional accreditation process and relevant accreditation standards under which third partiesparty auditors may be accredited for a three-year period, which may be extended through an appropriate process, to certify adherence by developers to the best practices and standards adopted pursuant to subparagraph (A).their requirements under Section 22603 of the Business and Professions Code.
(4) Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
(5) (A) Issue guidance describing the categories of artificial intelligence safety events that are likely to constitute a state of emergency within the meaning of subdivision (b) of Section 8558 and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
(B) The guidance issued pursuant to subparagraph (A) shall not limit, modify, or restrict the authority of the Governor in any way.
(6) Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
(7) Appoint and consult with an advisory committee for open-source artificial intelligence that shall do all of the following:
(A) Issue guidelines for model evaluation for use by developers of open-source artificial intelligence models that do not have hazardous capabilitieslack the ability to cause or enable critical harms.
(B) Advise the Frontier Model DivisionLegislature on the creation and feasibility of incentives, including tax credits, that could be provided to developers of open-source artificial intelligence models that are not covered models.
(C) Advise the Frontier Model Division on future policies and legislation impacting open-source artificial intelligence development.
(8) Levy fees, including an assessed fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administering this section that do not exceed the reasonable costs of administering this section.
(9) (A) Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of Section 22603 of the Business and Professions Code that the Judicial Council may, at its discretion, adopt.
(B) In developing the model jury instructions required by subparagraph (A), the Frontier Model Division shall consider all of the following factors:
(i) The level of rigor and detail of the safety and security protocol that the developer faithfully implemented while it trained, stored, and released a covered model.
(ii) Whether and to what extent the developer’s safety and security protocol was inferior, comparable, or superior, in its level of rigor and detail, to the safety and security protocols of comparable developers.
(iii) The extent and quality of the developer’s safety and security protocol’s prescribed safeguards, capability testing, and other precautionary measures with respect to the relevant hazardous capability and related hazardous capabilitiesrisk of causing a critical harm.
(iv) Whether and to what extent the developer and its agents complied with the developer’s safety and security protocol, and to the full degree, that doing so might plausibly have avoided causing a particular harm.
(v) Whether and to what extent the developer carefully and rigorously investigated, documented, and accurately measured, insofar as reasonably possible given the state-of-the-art, relevant risks that its model might pose.
(10) (A) On or before July 1, 2026, issue guidance regarding both of the following:
(10) (A) On or before Jan 1, 2027, and annually thereafter, issue regulations to update the definition of a “covered model” to ensure that it accurately reflects technological developments, scientific literature, and widely-accepted national and international standards and applies to artificial intelligence models that pose the greatest risk of enabling critical harms. The updated definition shall contain the following:
(i) Information relevant to determining whetherThe initial compute threshold that an artificial intelligence model ismust exceed to be considered a covered model, as defined in Section 22602 of the Business and Professions Code.
(ii) Technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption under paragraph (2) of subdivision (a) of Section 22603 of the Business and Professions Code.
(ii) The fine-tuning compute threshold that an artificial intelligence model must meet to be considered a covered model.
(B) In developing guidanceregulations pursuant to this paragraph, the Frontier Model Division shall take into account both of the following:
(i) The quantity of computing power used to train covered models that have been identified as having hazardous capabilitiesbeing reasonably likely to cause or not having hazardous capabilities when accounting forenable a reasonable margin for safetycritical harm.
(ii) Similar thresholds used in federal law, guidance, or regulations for the management of hazardous capabilitiesmodels with reasonable risks of causing critical harms.
(11) At least every(iii) Input from stakeholders, including academics, industry, and government entities, including from the open-source community.
(10) Every 24 months after initial publication of guidance under paragraphs (3), (5), and (10), review existing guidance in consideration of technological advancements, changes to industry best practices, and information received pursuant to paragraph (1) and update its guidance to the extent appropriate.
(1211) On and after January 1, 2026, annually publish the inflation-adjusted dollar amounts described in paragraph (3) of subdivision (n) and paragraph (2) of subdivision (f) of Section 22602 of the Business and Professions Code.
(df) There is hereby created in the General Fund the Frontier Model Division Programs Fund.
(1) All fees received by the Frontier Model Division pursuant to this section shall be deposited into the fund.
(2) All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this section.
[…]
ARGUMENTS IN SUPPORT:
The Center for AI Safety Action Fund, a co-sponsor of this bill, writes:
If California does not act to establish a clear and sensible governance framework, the safety of its citizens could be imperiled, the nation’s security could be seriously harmed, and AI’s enormous potential to improve our world could be derailed. Placing sensible guardrails around serious risks that the most powerful systems might pose, while taking significant steps towards leveling the playing field for academics and startups, is the best way to ensure that CA’s citizens can realize the immense benefits of this technology.
Encode Justice, a co-sponsor of this bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as “frontier AI models.” These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue. These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden’s Executive Order on Artificial Intelligence for their significant national security and public safety implications.
A coalition of labs, startups, and other entities including The Future Society write:
Building on Executive Orders on artificial intelligence from Governor Newsom and the Biden Administration, and the voluntary commitments to the White House, SB 1047 sets out clear standards for developers of the largest AI models, more powerful than any model that exists today — defined as those trained with more than 1026 floating-point operations of computing power and costing more than $100 million to train. SB 1047 will help ensure that these systems are developed in a safe and secure fashion while ensuring that California remains a leader in AI development.
ARGUMENTS IN OPPOSITION:
California Chamber of Commerce writes on behalf of a coalition of trade associations:
In addition to creating inconsistencies with federal regulations, the bill demands compliance with various vague and impractical, if not technically infeasible, requirements for which developers will be subject to harsh penalties, including potential criminal liability. We are concerned that the bill regulates AI technology as opposed to its high-risk applications, creates significant regulatory uncertainty and therefore high compliance costs, and poses significant liability risks to developers for failing to foresee and block any harmful use of their models by others – all of which inevitably discourages economic and technological innovation. And while recent amendments take important steps in responding to the open-source community, we remain concerned about the impact of the bill on AI research and development in California and the impact on startups. Overall, the bill still makes AI business too risky in California, particularly given the potential penalties under SB 1047.
The Chamber of Progress writes:
It is critical that public policy foster an abundance of frontier models - open and closed alike, existing and new entrants. A plurality of models will catalyze AI application development and ultimately benefit consumers. However, SB 1047 gives the largest incumbent AI models and models built upon them (“derivative models”) special treatment that will inevitably lead to fewer upstart (“non-derivative”) models. This will entrench the largest incumbent players in AI frontier model development - making them even more consequential - and undercut innovation when we should be encouraging a proliferation of approaches.
REGISTERED SUPPORT / OPPOSITION:
Support
Center for Ai Safety Action Fund (co-sponsor)
Economic Security Project Action (co-sponsor)
Encode Justice (co-sponsor)
Ae Studio
Ai Safety Student Team (HARVARD)
Apart Research
California State Council of Service Employees International Union (SEIU California)
Cambridge Boston Alignment Initiative
Causative Labs
Chapman University
Civic Ai Security Program
Denizen
Depict.ai
Elicit
Enh Alpha LLC
Far Ai, INC.
Fathom Radiant
General Agents
General Proximity
Gladstone Ai
Higher Ground Labs
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Loveable Labs Incorporated
MIT Ai Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe Ai Future
The Future Society
White Space Marketing Group
Support If Amended
Electronic Frontier Foundation
Oakland Privacy
Opposition
Acclamation Insurance Management Services
Allied Managed Care
Association of National Advertisers
California Chamber of Commerce
California Fuels and Convenience Alliance
California Land Title Association
California Manufacturers and Technology Association
Chamber of Progress
Civil Justice Association of California
Coalition of Small and Disabled Veteran Businesses
Computer & Communications Industry Association
Consumer Technology Association
Flasher Barricade Association
Insights Association
Los Angeles County Business Federation (BIZ-FED)
National Federation of Independent Business (NFIB)
Silicon Valley Leadership Group
Software and Information Industry Association
Technet
Oppose Unless Amended
BSA the Software Alliance
California Life Sciences
Analysis Prepared by: Slater Sharp / P. & C.P. / (916) 319-2200, Josh Tosney / P. & C.P. / (916) 319-2200
Future of Life Institute, “Asilomar AI Principles,” Aug. 11, 2017, https://futureoflife.org/open-letter/ai-principles/. ↑
National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework,” Jan. 2023, https://doi.org/10.6028/NIST.AI.100-1. ↑
The White House, “Blueprint for an AI Bill of Rights,” Oct. 2023, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. ↑
TechEquity, “AI Policy Principles,” Mar. 2024, https://techequitycollaborative.org/wp-content/uploads/2024/03/AI_Policy_Principles.pdf. ↑
Future of Life Institute, “Pause Giant AI Experiments: An Open Letter,” Mar. 22, 2023, https://futureoflife.org/wp-content/uploads/2023/05/FLI_Pause-Giant-AI-Experiments_An-Open-Letter.pdf. ↑
Center for AI Safety, “Statement on AI Risk,” May 30, 2023, https://www.safe.ai/work/statement-on-ai-risk#open-letter. ↑
Matthias Plappert, Durk Kingma, Max Chen, Cage Zhong, and Penny Deng, “Thoughts on Llama 3,” Factorial Funds, Apr. 24, 2024, https://www.factorialfunds.com/blog/thoughts-on-llama-3. ↑
European Union, “EU Artificial Intelligence Act,” https://artificialintelligenceact.eu/article/51/. ↑
Yam Peleg, “GPT-4's details are leaked,” Twitter, Jul, 11, 2023, https://archive.is/2RQ8X#selection-463.1-463.28. ↑
OpenAI, “Introducing GPT-4o: our fastest and most affordable flagship model,” https://platform.openai.com/docs/guides/fine-tuning. ↑
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 1047 |
THIRD READING
Bill No: SB 1047
Author: Wiener (D), et al.
Amended: 5/16/24
Vote: 21
SENATE JUDICIARY COMMITTEE: 9-0, 4/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Min, Stern, Wahab
NO VOTE RECORDED: Wilk, Niello
SENATE GOVERNMENTAL ORG. COMMITTEE: 11-0, 4/23/24
AYES: Dodd, Wilk, Ashby, Bradford, Glazer, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Smallwood-Cuevas
NO VOTE RECORDED: Alvarado-Gil, Archuleta, Jones, Nguyen, Seyarto
SENATE APPROPRIATIONS COMMITTEE: 5-0, 5/16/24
AYES: Caballero, Ashby, Becker, Bradford, Wahab
NO VOTE RECORDED: Jones, Seyarto
SUBJECT: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
SOURCE: Center for AI Safety Action Fund
Economic Security Project Action
Encode Justice
DIGEST:
This bill requires developers of powerful artificial intelligence models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. This bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computing cluster.
ANALYSIS:
Existing law:
- Establishes the California Department of Technology (CDT) within the Government Operations Agency, under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Gov. Code Sec. 11545(a).)
- Provides that persons are responsible, not only for the result of their willful acts, but also for an injury occasioned to another by their want of ordinary care or skill in the management of their property or person, except so far as the latter has, willfully or by want of ordinary care, brought the injury upon themselves. (Civ. Code § 1714(a).)
This bill:
- Establishes the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
- Provides definitions for all relevant terms:
- “Covered model” means an artificial intelligence model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations (FLOP) or a model that can reasonably be expected to have similar performance capabilities as assessed by commonly used benchmarks.
- “Limited duty exemption” means an exemption with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
- “Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
- At least $500 million of damage through cyberattacks on critical infrastructure.
- At least $500 million of damage by a model that autonomously engages in conduct that would be criminal if done by a human.
- Other comparably severe threats to public safety and security.
- “Hazardous capability” includes the capabilities above even if they would not manifest but for fine tuning and posttraining modifications performed by third-party experts.
- “Fine tuning” means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
- “Covered guidance” means either of the following:
- Guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division that is relevant to the management of safety risks associated with artificial intelligence models that may possess hazardous capabilities.
- Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with AI models that may possess hazardous capabilities.
- “Computing cluster” means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
- Establishes the Frontier Model Division (FMD) within CDT and tasks them with various duties, including:
- Review annual certification reports from developers.
- Advise the Attorney General on potential violations of this law.
- Issue guidance, standards, and best practices necessary to prevent unreasonable risks from covered models with hazardous capabilities.
- Requires developers of covered models that are not subject to a limited duty exemption, before training the model and until an exemption establishes, to implement various safeguards. This includes:
- Implements administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, that are appropriate given the associated risks.
- Implements the capability to enact a full shutdown or a “kill switch.”
- Implements all covered guidance.
- Implements, and annually reviews, a written and separate safety and security protocol, as specified, that details how it is adequate to prevent critical harms and “hazardous capabilities” and specifies the testing procedures incorporated therein. The protocol shall be provided to FMD.
- Ensures that these protocol are implemented as written, including by designating senior personnel responsible and conducting audits, as appropriate.
- Refrains from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
- Establishes specified processes for and imposes obligations on developers throughout model training and the initiation of widespread use of the models, as provided, including requiring specified reporting and implementation of appropriate safeguards.
- Requires persons that operate computing clusters to implement appropriate written policies and procedures to do the following when a customer utilizes compute resources that would be sufficient to train a covered model:
- Obtains, and annually validates, a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster, as specified.
- Annually assesses whether a prospective customer intends to utilize the cluster to deploy a covered model.
- Maintains for seven years and provides to the Frontier Model Division or the Attorney General, records of actions taken pursuant to this law.
- Implements the ability to promptly enact a full shutdown in the event of an emergency.
- Authorizes the Attorney General to bring an action seeking recovery of preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for the violation, including deletion of the covered model and the weights utilized in that model, as provided. Monetary damages, including punitive damages, to persons aggrieved and a court order for a full shutdown are also available. However, these remedies are only available in response to harm or an imminent risk or threat to public safety. The Attorney General may also recover a civil penalty in an amount not exceeding 10 percent of the cost, excluding labor, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor, to develop the covered model for any subsequent violation. Specified remedies are not available until six months to one year after enactment.
- Clarifies that the duties and obligations imposed are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
- Tasks CDT with creating a public cloud computing cluster known as CalCompute through the commissioning of consultants with specified objectives, first of which is to study the safe and secure deployment of large-scale AI models.
- Authorizes CDT to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate the establishment of CalCompute.
Background
Owing to recent advances in processing power and the rise of big data, artificial intelligence’s (AI) capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact business, and consume media. It has been used to accelerate productivity, achieve efficiencies, liberate us from drudgery, write our college essay, help us understand and enjoy the world, upgrade the Pope’s fashion, connect with each other, and live longer, fuller lives. It has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy. For good or ill, its transformative potential seems boundless.
This bill seeks to establish guardrails for the development of the most powerful AI models to avoid the more catastrophic possibilities about which experts have raised alarms. It places a series of obligations on developers of “covered models” and providers of the cloud compute for training such models. This bill also seeks to establish a public cloud-computing cluster that facilitates equitable participation in the development and deployment of responsible AI systems. This bill is co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. It is supported by a host of tech companies and labor organizations, as well as the Los Angeles Area Chamber of Commerce, and opposed by the Chamber of Progress and a coalition of industry associations.
Comments
According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a public AI research cluster that will allow startups, researchers, and community groups to participate in the development of large-scale AI systems. By providing a broad range of stakeholders with access to the AI development process, CalCompute will help align large-scale AI systems with the values and needs of California communities.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes
According to the Senate Appropriations Committee:
- CDT indicates significant one-time costs and ongoing costs (General Fund) to implement this bill.
- Judicial Council estimates minor and absorbable costs.
- Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to the courts.
- Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill.
SUPPORT: (Verified 5/17/24)
Center for AI Safety Action Fund (co-source)
Economic Security Project Action (co-source)
Encode Justice (co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program
Denizen
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Elicit
Enh Alpha LLC
Far AI, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group
OPPOSITION: (Verified 5/17/24)
Association of National Advertisers
California Chamber of Commerce
California Manufacturers and Technology Association
Chamber of Progress
Civil Justice Association of California
Computer and Communications Industry Association
Insights Association
Software and Information Industry Association
Technet
ARGUMENTS IN SUPPORT: Encode Justice, a co-sponsor of the bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as “frontier AI models.” These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue.
These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden’s Executive Order on Artificial Intelligence for their significant national security and public safety implications.
ARGUMENTS IN OPPOSITION: Chamber of Progress writes:
Unfortunately, SB 1047 forces model developers to engage in speculative fiction about imagined threats of machines run amok, computer models spun out of control, and other nightmare scenarios for which there is no basis in reality.
Instead, SB 1047 forces developers operating in the real world to proactively mitigate against every conceivable harm - and many inconceivable ones - not just by the model itself, but subsequent third parties who make use of the model.
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
5/20/24 18:52:26
**** END ****
SENATE COMMITTEE ON APPROPRIATIONS
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
SB 1047 (Wiener) - Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
Version: April 30, 2024 | Policy Vote: JUD. 9 - 0, G.O. 11 - 0 |
Urgency: No | Mandate: Yes |
Hearing Date: May 16, 2024 | Consultant: Liah Burnley |
Bill Summary:
SB 1047 requires developers of artificial intelligence (AI) models to implement safeguards and policies to prevent specified hazards and threats to public safety and authorizes the Attorney General (AG) to enforce these provisions. SB 1047 also establishes the Frontier Model Division within the Department of Technology (CDT) to oversee the development of AI models and requires CDT to commission consultants to create a public cloud computing cluster “CalCompute,” to conduct research into the safe and secure deployment of large-scale AI models.
*********** ANALYSIS ADDENDUM – SUSPENSE FILE ***********
The following information is revised to reflect amendments
adopted by the committee on May 16, 2024
Fiscal Impact:
- CDT indicates significant one-time costs and ongoing costs (General Fund) to implement this bill.
- Judicial Council estimates minor and absorbable costs.
- Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to the courts.
- Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill.
Author Amendments:
Delete several provisions related to responsibilities of the Frontier Model Division, as specified.
-- END --
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
SB 1047 (Wiener) - Safe and Secure Innovation for Frontier Artificial Intelligence Models Act
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
SB 1047 requires developers of artificial intelligence (AI) models to implement safeguards and policies to prevent specified hazards and threats to public safety and authorizes the Attorney General (AG) to enforce these provisions. SB 1047 also establishes the Frontier Model Division within the Department of Technology (CDT) to oversee the development of AI models and requires CDT to commission consultants to create a public cloud computing cluster “CalCompute,” to conduct research into the safe and secure deployment of large-scale AI models.
Fiscal Impact:
- CDT indicates significant one-time costs and ongoing costs (General Fund) to implement this bill. According to CDT one-time costs estimated over $7,000,000, to recruit and retain staff and external consultants and ongoing costs estimated over $9,000,000 to staff 24 positions and external consultants. CDT notes that this bill would require technical, analytic, administrative, operational, programming, research, and design work. CDT would need to establish a new team focused solely on building out this new division and begin to define, scope, and implement initial stages of CalCompute. As a result, CDT estimates that it would need 26 general fund positions, two temporary program and project managers to design, develop, and implement this new division, with 24 ongoing positions to adequately and sustainably create, manage, and secure the Frontier Model Division and CalCompute. CDT notes the importance of integrating in-house talent to ensure proper oversight in responsible use of AI across strategy, design, and technical functions. These foundational estimates include primarily full-time employees, with an additional request for funding to source and award contracts for external consultants necessary to provide expertise in the short-term within defined scopes of work to be determined by CDT. In parallel, a recruitment strategy specifically built to attract and retain talent is necessary to meet increased demand for AI expertise within and outside of the CDT.
- Judicial Council estimates minor and absorbable costs within existing resources to review and process the model jury instructions recommended by the Frontier Model Division, as required by this bill. According to Judicial Council, the language in this bill allows sufficient flexibility for the Judicial Council to utilize existing procedures including an advisory committee review, public comment period, and approval by the Judicial Council prior to official adoption and dissemination of any new jury instructions.
- Unknown workload cost pressures (General Fund, Trial Court Trust Fund) to the courts to adjudicate potential civil enforcement actions for violations of this bill brought by the AG. The fiscal impact of this bill to the courts will depend on many unknown factors, including the numbers of violations alleged to have occurred, if parties settle the matter before the filing of an action, and the factors unique to each case. While it is not known how many actions would be filed resulting from the various changes that this measure would make, it generally costs about $8,300 to operate a courtroom for one eight-hour day. Consequently, if reforms proposed by this bill lead to the filing of cases that, combined, take 50 or more hours of court involvement, the cost pressures of this measure to the courts would be significant. This bill would further impact to the courts to adjudicate criminal perjury charges brought against defendants who violate the bill’s perjury provisions. A defendant is entitled to a no cost legal representation and a jury trial. If only four new perjury charge is filed annually statewide and proceed to trial resulting in the use of two days of court time the approximate cost to the trial courts is $66,400. While the superior courts are not funded on a workload basis, an increase in workload could result in delayed court services and pressure to fund additional staff and resources. The Governor’s 2024-25 state budget proposes $83.1 million ongoing General Fund to continue to backfill the Trial Court Trust Fund for expected revenue declines.
- Potential non-reimbursable annual costs (local funds, General Fund) in the hundreds of thousands of dollars to counties for increased incarceration costs relating to the expansion of felony perjury in this bill. Actual incarceration costs will depend on the number of convictions, the length of each sentence. The average annual cost to incarcerate one person in county jail is approximately $29,000. Based on the average annual cost to incarcerate a defendant in county jail, if one defendant statewide is sentenced annually to an average of three years in county jail for perjury, the total cost to counties would be $87,000. California county jails are increasingly overcrowded – particularly since the enactment of the Realignment Act in 2011. Although new crimes are not considered reimbursable state mandates pursuant to Proposition 30 (2012), overcrowding in county jails creates cost pressure on the General Fund because the state has historically granted new funding to counties to offset overcrowding since 2011.
Background:
In 2018, the Little Hoover Commission released a report titled, “Artificial Intelligence: A Roadmap for California” which detailed the possible benefits and risks of AI. The report encouraged the Governor and the Legislature to: engage the public in broad conversations about the responsible use of AI; identify the necessary training and other educational opportunities to build the human infrastructure required for an AI-fueled economy; explore opportunities to form and leverage public private partnerships to use AI to address societal challenges; and examine legislative and policy strategies to preserve and protect key public values, such as privacy, transparency and accountability.
Proposed Law:
This bill does all of the following:
- Includes numerous definitions, including, among others:
- “AI model” means an engineered or machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
- “Computing cluster” means a set of machines connected by data center networking that has a specified computing capacity and can be used for training AI.
- “Covered model” means an AI model that was trained using a specified quantity of computing power.
- “Developer” means a person or entity that creates, owns, or otherwise has responsibility for an AI model.
- “Limited duty exemption” means an exemption, with respect to a covered model that a developer can reasonably exclude the possibility has a hazardous capability or may come close to possessing a hazardous capability, as specified.
- “Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties;
- At least five hundred million dollars $500,000,000 of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents;
- At least $500,000,000 of damage by an AI model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human; or,
- Other threats to public safety and security that are of comparable severity.
- Provides that, before initiating training of a covered model, a developer may determine whether the covered model qualifies for an exemption, as specified. Upon determining that a covered model qualifies for an exemption, the developer must submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination.
- Provides that, before initiating training of a covered model that and is not the subject of an exemption, and until that covered model is the subject of an exemption, the developer of that covered model shall implement protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, as specified.
- Provides that, upon completion of the training of a covered model that is not the subject of an exemption, the developer shall perform capability testing sufficient to determine if an exemption applies. Upon determining if an exemption applies the developer shall submit to the Frontier Model Division, under penalty of perjury, a certification of compliance within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model, as specified.
- Provides that, before initiating the commercial, public, or widespread use of a covered model that is not subject to an exemption, the developer must implement reasonable safeguards and requirements, as specified.
- Requires a developer of a covered model that is not the subject of an exemption to submit to the Frontier Model Division an annual certification under penalty of perjury of compliance signed by the chief technology officer, or a more senior corporate officer.
- Requires developers to report all AI safety incidents affecting a covered model to the Frontier Model Division in a manner prescribed by the Frontier Model Division.
- Requires a developer of a covered model that provides commercial access to that covered model to provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model.
- Requires a person who operates a computing cluster to implement written policies and procedures when a customer utilizes compute resources that would be sufficient to train a covered model, as specified.
- Requires a person that operates a computing cluster to provide a transparent, uniform, publicly available price schedule for the purchase of access to the computing cluster.
- Authorizes the Attorney General to commence civil proceedings to enforce these provisions and requires the court to disregard corporate formalities and impose joint and several liability on affiliated entities, as specified.
- Creates the Frontier Model Division within CDT to do all of the following:
- Compile annual certification reports received from developers and publicly release summarized findings based on those reports;
- Advise the AG on potential violations of the provisions in this bill;
- Issue guidance, standards, and best practices necessary to prevent unreasonable risks from covered models with hazardous capabilities;
- Establish an optional accreditation process and relevant accreditation standards;
- Publish anonymized AI safety incident reports;
- Appoint and consult an advisory committee that shall, among other things, advise the Governor on when it may be necessary to proclaim a state of emergency relating to AI and advise the Governor on what responses may be appropriate in that event and to provide technical assistance and advice to the Legislature, upon request, with respect to AI-related legislation;
- Levy fees, including a fee for the submission of a certification, in an amount sufficient to cover the reasonable costs of administration; and,
- Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations provisions of this bill, that the Judicial Council may, at its discretion, adopt develop guidance to developers regarding development of AI models, as specified.
- Creates the Frontier Model Division Programs Fund within the General Fund. All fees received by the Frontier Model Division shall be deposited into the fund. All moneys in the account shall be available, only upon appropriation by the Legislature, for purposes of carrying out the provisions of this bill relating to the Frontier Model Division.
- Establishes CalCompute, a public cloud computing cluster, within CDT with the primary focus of conducting research into the safe and secure deployment of AI and fostering equitable innovation.
- Requires CDT to commission consultants for CalCompute that are representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders to meet specified objectives.
- Requires CDT to submit an annual report to the Legislature from the commissioned consultants to ensure progress in meeting specified objectives.
- Provides that CDT may receive private donations, grants, and local funds, in addition to allocated funding in the annual budget.
Related Legislation:
- SB 970 (Ashby, 2024) requires Judicial Council to review the impact of AI on court proceedings and develop rules of court for claims that evidence has been generated by or manipulated by AI. AB 970 is pending in this Committee.
- SB 942 (Becker, 2024) establishes the California AI Transparency Act, which, among other things, requires a provider, to create an AI detection tool by which a person can query the covered provider as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a generative AI system, as defined, provided by the covered provider that meets certain criteria. AB 942 is pending in this Committee.
- AB 2930 (Bauer-Kahan, 2024) requires, among other things, a developer of an automated decision tool to perform an impact assessment of the automated decision tool. AB 2930 is pending in Assembly Appropriations Committee.
- AB 3211 (Wicks, 2024) establishes the California Provenance, Authenticity and Watermarking Standards Act, which requires an AI system provider, as defined, to assist in the disclosure of data to mitigate harms caused by inauthentic content. AB 3211 is pending in Assembly Appropriations Committee.
- AB 1576 (Calderon, 2019) would have required the Secretary of Government Operations to appoint participants to an AI working group to evaluate the uses, risks, benefits, and legal implications associated with the development and deployment of AI by California-based businesses. AB 1576 was held in this Committee.
- SB 348 (Chang, 2019) would have require CDT to devise a strategic plan designed to aid departments and agencies with the incorporation of AI into state information technology strategic plans, policies, standards, and enterprise architecture to improve state services. SB 348 failed passage in Assembly Appropriations Committee.
- SB 752 (Stern, 2019) would have created the Commission on Tech Equity to convene a public process to gather input and to understand the economic, social, workplace, and technological landscape of innovation and technology in California. SB 752 was held in the Senate Appropriations Committee.
- AB 594 (Salas, 2019) would have required the Director of Technology to appoint a Chief AI Officer within the CDT to evaluate the uses of AI in state government and to advise the Director on incorporating AI into state information technology strategic plans, policies, standards, and enterprise architecture. AB 594 was vetoed.
Staff Comments:
This bill is one of a handful of bills related to AI technology in this Legislative Session. While all these bills are not focused on the same aspects of AI technology, many of them would require duplicative work and could result in inconsistent recommendations to the Legislature.
-- END --
SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION
Senator Bill Dodd
Chair
2023 - 2024 Regular
Bill No: SB Hearing Date: 4/23/2024
Author: | Wiener, et al. | ||
Version: | Amended | ||
Urgency: | Fiscal: | ||
Consultant: | |||
SUBJECT:
DIGEST: This bill, the Safe and Secure Innovation for Frontier Artificial Intelligence (AI) Systems Act, requires developers of powerful AI models and those providing the computing power to train such models to place appropriate safeguards and policies to prevent critical harms, as specified. Additionally, this bill creates the Frontier Model Division within the California Department of Technology (CDT) to oversee the development of those models and requires CDT to create a public cloud computing cluster, to be known as CalCompute, as specified.
ANALYSIS:
Existing law:
- Establishes CDT, within the Government Operations Agency (GovOps), and requires CDT to, among other things, identify, assess, and prioritize high-risk, critical information technology services and systems across state government for modernization, stabilization, or remediation.
- Provides that persons are responsible, not only for the result of their willful acts, but also for an injury occasioned to another by their want of ordinary care or skill in the management of their property or person, except so far as the latter has, willfully or by want of ordinary care, brought the injury upon themselves.
- The Emergency Services Act authorizes the Governor to declare a state of emergency, and local officials and local governments to declare a local emergency, when specified conditions of disaster or extreme peril to the safety of persons and property exist.
- Defines the term “state of emergency” and “local emergency” to mean a duly proclaimed existence of conditions of disaster or of extreme peril to the safety of persons and property within the state caused by, among other things, fire, storm, riot, or cyberterrorism, as specified.
- Requires a report that is required or requested by law to be submitted by a state or local agency to the Members of either house of the Legislature to instead be submitted as a printed copy to the Secretary of the Senate, as an electronic copy to the Chief Clerk of the Assembly, and as an electronic or printed copy to the Legislative Counsel. Each report shall include a summary of its contents, not to exceed one page in length.
This bill:
- Establishes the Safe and Secure Innovation for Frontier AI Systems Act, and provides definitions for relevant terms, including:
- “Advanced persistent threat” means an adversary with sophisticated levels of expertise and significant resources that allow it, through the use of multiple different attack vectors, as specified.
- “AI model” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
- “Computing cluster” means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations (FLOP) per second and can be used for training AI.
- “Covered guidance” means guidance issued by the National Institute of Standards and Technology (NIST) and by the Frontier Model Division that is relevant to the management of safety risks associated with AI models that may possess hazardous capabilities or industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with AI models that may possess hazardous capabilities.
- “Covered model” means an AI model that was trained using a quantity of computing power greater than 10^26 integer or FLOP or a model that can reasonably be expected to have similar performance capabilities as assessed by commonly used benchmarks, as specified.
- “Critical infrastructure” means assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state.
- “Derivative model” means an AI model that is a derivative of another AI model, including a modified or unmodified copy of an AI model or a combination of an AI model with other software. Derivative model does not include an entirely independently trained AI model.
- “Full shutdown” means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person, including any computer or storage device remotely provided by agreement.
- “Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
- At least $500 million of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.
- At least $500 million of damage by an AI model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.
- Other threats to public safety and security that are of comparable severity to the harms described above.
- “Limited duty exemption” means an exemption with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of post-training modifications.
- “Model weight” means a numerical parameter established through training in an AI model that helps determine how input information impacts a model’s output.
- “Open-source AI model” means an AI model that is made freely available and may be freely modified and redistributed.
- “Post-training modification” means the modification of the capabilities of an AI model after the completion of training by any means, including, but not limited to, initiating additional training, providing the model with access to tools or data, removing safeguards against hazardous misuse or misbehavior of the model, or combining the model with, or integrating it into, other software.
- “Safety and security protocol” means documented technical and organizational protocols that are used to manage the risks of developing and operating covered models across their life cycle, including risks posed by enabling or potentially enabling the creation of derivative models or the protocols specify that compliance with the protocols is required in order to train, operate, possess, and provide external access to the developer’s covered model.
- Creates the Frontier Model Division, within CDT, and requires the division to do all of the following:
- Review annual certification reports received from developers and publicly release summarized findings based on those reports, as specified.
- Advise the Attorney General (AG) on potential violations of this bill, as specified.
- Issue guidance, standards, and best practices, as specified, and establish an optional accreditation process and relevant accreditation standards, as specified.
- Publish anonymized AI safety incident reports received from developers, as specified.
- Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices, as specified.
- Issue guidance describing the categories of AI safety events that are likely to constitute a state of emergency, as specified, and responsive actions that could be ordered by the Governor after a duly proclaimed state of emergency.
- Appoint and consult with an advisory committee to advise the Governor on when it may be necessary to proclaim a state of emergency relating to AI and advise the Governor on what responses might be appropriate in that event.
- Appoint and consult with an advisory committee for open-source AI to issue guidelines for model evaluation and advise the Frontier Model Division, as specified.
- Provide technical assistance and advice to the Legislature, upon request, with respect to AI-related legislation.
- Monitor relevant developments relating to the safety risks associated with the development of AI models and the functioning of markets for AI models.
- Levy fees, including an assessed fee for the submission of a certification, as specified.
- Develop and submit to the Judicial Council proposed model jury instructions for actions involving violations of this bill that the Judicial Council may, at its discretion, adopt, as specified.
- On or before July 1, 2026, issue guidance regarding technical thresholds and benchmarks relevant to determining whether an AI model is a covered model, as specified, whether the covered model is subject to a limited duty exemption, as specified.
- At least every 24 months after initial publication of guidance, review existing guidance in consideration of technological advancements, changes to industry best practices, and information received, as specified.
- Creates, in the General Fund, the Frontier Model Division Programs Fund, for all fees received by the Frontier Model Division, and makes those fees, subject to appropriation by the Legislature, available for the purposes of carrying out the provisions of this bill, as specified.
- Authorizes a developer to determine that a covered model qualifies for a limited duty exemption if the covered model will have lower performance on all benchmarks relevant per this bill and does not have a greater general capability than a noncovered model that manifestly lacks hazardous capabilities or another model that is the subject of a limited duty exemption, as specified.
- Requires a developer, upon determining that a covered model qualifies for a limited duty exemption, to submit to the Frontier Model Division a certification under penalty of perjury that specifies the basis for that determination, as specified.
- Provides that a developer that makes a good faith error regarding a limited duty exemption will be deemed to be in compliance with this bill if the developer reports its error to the Frontier Model Division within 30 days of completing the training of the covered model and ceases operation of the AI model until the developer is otherwise in compliance, as specified.
- Requires a developer, before initiating training of a covered model that is not a derivative model and is not the subject of a limited duty exemption, to do all of the following:
- Implement administrative, technical, and physical cybersecurity protections, as specified.
- Implement the capability to promptly enact a full shutdown of the covered model.
- Implement all covered guidance.
- Implement a written and separate safety and security protocol, as specified.
- Ensure that the safety and security protocol is implemented as writing, as specified.
- Provide a copy of the safety and security protocol to the Frontier Model Division.
- Conduct an annual review of the safety and security protocol to account for any changes to the capabilities of the covered model and industry best practices and, if necessary, make modifications to the policy.
- If the safety and security protocol is modified, provide an update copy to the Frontier Model Division within 10 business days.
- Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
- Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, NIST, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
- Requires the developer of a non-derivative covered model that is not the subject of a limited duty exemption, upon completion of the training of the covered model, to perform capability testing sufficient to determine whether the developer can make a positive safety determination with respect to the covered model pursuant to its safety and security protocol.
- Requires a developer, upon making a limited duty exemption with respect to a covered model, to submit to the Frontier Model Division a certification under penalty of perjury of compliance with the requirements of this bill within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model that includes the basis for the developer’s determination that of a limited duty exemption and the specific methodology and results of the capability testing undertaken pursuant to this bill.
- Requires a developer, before initiating the commercial, public, or widespread use of a covered model that is not subject to a limited duty exemption, to do all of the following:
- Implement reasonable safeguards and requirements, as specified.
- Provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm.
- Refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
- Implement other measures that are reasonably necessary, including in light of applicable guidance from the Frontier Model Division, NIST, and standard-setting organizations, to prevent the development or exercise of hazardous capabilities or to manage the risks arising from them.
- Requires the developer of a covered model to periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented pursuant to this section in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass those procedures, policies, protections, capabilities, and safeguards.
- Requires a developer of a non-derivative covered model that is not the subject of a limited duty exemption to submit to the Frontier Model Division an annual certification under penalty of perjury of compliance with the requirements of this bill signed by the chief technology officer, or a more senior corporate officer, as specified.
- Requires the certification to specify or provide, at a minimum, the nature and magnitude of hazardous capabilities, as specified, an assessment of the risk, as specified, and other information useful to accomplishing the purposes of this bill, as determined by the Frontier Model Division.
- Requires a developer to report each AI safety incident affecting a covered model to the Frontier Model Division, as specified.
- Specifies that reliance on an unreasonable limited duty exemption does not relieve a developer of its obligations under this bill, as specified.
- Requires a person operating a computing cluster to implement appropriate written policies and procedures to do all of the following when a customer utilizes compute resources that would be sufficient to train a covered model:
- Obtain a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster, as specified.
- Assess whether a prospective customer intends to utilize the computing cluster to deploy a covered model.
- Annually validate the information collected pursuant to this bill and conduct the assessment required.
- Maintain for seven years and provide to the Frontier Model Division or the Attorney General (AG), upon request, appropriate records of actions taken under this bill, including policies and procedures put into effect.
- Implement the capability to promptly enact a full shutdown in the event of an emergency.
- Requires the developer of a covered model that provides commercial access to that covered model and a person that operates a computing cluster to provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developer’s terms of service and prohibits the developer from engaging in unlawful discrimination or noncompetitive activity in determining price or access, as specified.
- Authorizes the AG, if the AG finds that a person is violating this bill, to bring a civil action, as specified.
- Authorizes the court, in a civil action under this bill, to award preventative relief, including a permanent or temporary injunction, restraining order, or other order as specified, or any other relief as the court deems appropriate, as specified.
- Requires a court to disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of this bill if the court concludes that steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability and the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this bill.
- Prohibits a developer from preventing an employee from disclosing information to the AG or from retaliating against the employee if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance with the requirements of this bill, as specified.
- Authorizes the AG to publicly release any complaint, or a summary of that complaint, pursuant to this bill if the AG concludes that doing so will serve the public interest.
- Requires a developer to provide clear notice to all employees working on covered models of their rights and responsibilities under this bill.
- Requires developers to provide a reasonable internal process through which an employee may anonymously disclose information to the developer if the employee believes in good faith that the information indicates that the developer is out of compliance with this bill or has made false or materially misleading statements related to its safety and security protocol, as specified.
- Requires CDT to commission consultants, as specified, to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale AI models and fostering equitable innovation that includes, but is not limited to, all of the following:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate use of CalCompute.
- Requires CDT, in order to meet the objective of establishing CalCompute, to require consultants commissioned to work on this process to evaluate and incorporate all of the following considerations into its plan:
- An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
- The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
- A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California, as specified.
- An analysis of the state’s investment in the training and development of the technology workforce, including through degree programs at the University of California (UC), the California State University (CSU), and the California Community Colleges (CCC).
- A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
- Requires CDT to submit an annual report to the Legislature from the commissioned consultants to ensure progress in meeting the objectives listed above, as specified.
- Authorizes CDT to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate this bill.
- Specifies that the provisions of this bill are severable, as specified.
Background
Author Statement. According to the author’s office, “large-scale AI has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.”
Further, “SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a public AI research cluster that will allow startups, researchers, and community groups to participate in the development of large-scale AI systems. By providing a broad range of stakeholders with access to the AI development process, CalCompute will help align large-scale AI systems with the values and needs of California communities.”
Defining AI. Reminiscent of the wave of legislation to regulate social media platforms, there are dozens of bills currently making their way through the Legislature to regulate AI. Before potentially subjecting the industry and regulators to a host of new laws, it seems imperative to first define exactly what it is we are talking about and to harmonize the definitions being used just as was done when the first comprehensive definition of “social media platform” was codified a few years ago.
The task is not necessarily a straightforward one, as pointed out by the United States Congressional Research Service:
Defining AI is not merely an academic exercise, particularly when drafting legislation. AI research and applications are evolving rapidly. Thus, congressional consideration of whether to include a definition for AI in a bill, and if so how to define the term or related terms, necessarily include attention to the scope of the legislation and the current and future applicability of the definition. Considerations in crafting a definition for use in legislation include whether it is expansive enough not to hinder the future applicability of a law as AI develops and evolves, while being narrow enough to provide clarity on the entities the law affects. Some stakeholders, recognizing the many challenges of defining AI, have attempted to define principles that might help guide policymakers. Research suggests that differences in definitions used to identify AI-related research may contribute to significantly different analyses and outcomes regarding AI competition, investments, technology transfer, and application forecasts.
There are several leading definitions of AI that can be relied on to craft a reasonably clear yet appropriately broad definition for this advancing technology. While there is no definition of AI in state law, the California Privacy Protection Agency (CPPA) has published draft regulations that provide a definition for AI:
“Artificial intelligence” means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The artificial intelligence may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different artificial intelligence varies in its levels of autonomy and adaptiveness after deployment. For example, artificial intelligence includes generative models, such as large language models, that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
At the federal level, the National Artificial Intelligence Act of 2020 provides the following:
Artificial intelligence. The term “artificial intelligence” means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. AI systems use machine and human-based inputs to—
(A) perceive real and virtual environments;
(B) abstract such perceptions into models through analysis in an automated manner; and
(C) use model inference to formulate options for information or action.
Also at the federal level, NIST, in their Artificial Intelligence Risk Management Framework (AI RMF 1.0), defined an AI system as “an engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.”
At the international level, the Organization for Economic Co-operation and Development (OECD) published and then revised a definition for AI to serve as a standard across jurisdictions:
An AI system is a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. Different AI systems vary in their levels of autonomy and adaptiveness after deployment.
With the revision, OECD made the case for harmonization:
Obtaining consensus on a definition for an AI system in any sector or group of experts has proven to be a complicated task. However, if governments are to legislate and regulate AI, they need a definition to act as a foundation. Given the global nature of AI, if all governments can agree on the same definition, it allows for interoperability across jurisdictions.
Most recently, the European Parliament signed the EU AI Act, which defines AI as follows:
“AI system” means a machine-based system designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.
In order to gain both the benefit of the expertise and compromise that went into formulating these definitions and the efficiencies that come with harmonization, the Senate Committee on Judiciary, the Senate Committee on Governmental Organization, and a variety of stakeholders, including the author, have come up with the following definition to begin this process and to include a definition of AI in the bill, which incorporates elements of the NIST definition into the internationally recognized formulations while eliminating unnecessary examples from within it:
“Artificial intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
Frameworks for Responsible Development and Accountability in AI. With recent dramatic advances in the capabilities of AI systems, the need for frameworks for accountability and responsible development have become ever more urgent.
In January of 2017, AI researchers, economists, legal scholars, ethicists, and philosophers met in Asilomar, California to discuss principles for managing the responsible development of AI. The collaboration resulted in the Asilomar Principles. Aspirational rather than prescriptive, these 23 principles were intended to initiate and frame a dialogue by providing direction and guidance for policymakers, researchers, and developers. Its endorsers include 1,200 leading experts in the field of AI, including DeepMind founder Demis Hassabis and the late Stephen Hawking.
As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.
While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed. These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.
The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.
The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.
Secondly, the Blueprint includes “Algorithmic Discrimination Protections.” Algorithmic discrimination occurs when automated systems contribute to unjustified different treatment or impacts disfavoring people based on their race, color, ethnicity, sex (including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual orientation), religion, age, national origin, disability, veteran status, genetic information, or any other classification protected by law. Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way.
Additionally, the Blueprint discusses “Data Privacy,” stating that individuals should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used.
Further, the Blueprint identifies “Notice and Explanation.” Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
The final principle identified by the Blueprint is “Human Alternatives, Consideration, and Fallback.” The Blueprint states that you should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts. In some cases, a human or other alternative may be required by law.
California Privacy Protection Agency (CPPA) Actions. In March of this year, the CPPA voted to move forward with draft automated decision system (ADS) technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students.
According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”
Further, CalMatters states that under the proposed rules, “businesses must notify people before using AI. If people opt out of interacting with an AI model, businesses cannot discriminate against people for that choice. If people agree to use an AI service or tool, businesses must respond to requests by individuals about how they use their personal information to make predictions. The rules would also require employers or third-party contractors to carry out risk assessments to evaluate the performance of their technology.”
According to the CPPA, the proposed framework includes the following three key components:
- Pre-use Notice Requirements. A business must provide information to the consumer about how it proposes to use the ADS so that the consumer can decide whether to opt-out or proceed, and whether to exercise their access right.
- Opt-out Right Requirements. For opt-out requests submitted before the business initiates the processing, the business must not process consumer’s personal information using that ADS. For opt-out requests submitted after the business has initiated the processing, the business must cease processing the consumer’s personal information using that ADS and notify relevant entities of the opt-out and instruct them to comply.
- Access Right Requirements. This involves the purpose of using the ADS; the output with respect to the consumer; how the businesses used the output with respect to the consumer; how the ADS worked with respect to the consumer; and that the businesses cannot retaliate against consumers for exercising CPPA rights.
The CPPA will spend the rest of the spring and summer performing a statewide informational outreach tour before initiating formal rulemaking, which is expected later this year.
Safe and Secure Innovation for Frontier AI Systems Act. This bill takes a number of approaches in responding to concerns in AI regulation. The bill can generally be broken up into the following elements:
- Safety and security obligations imposed on developers of powerful AI models.
- Know-your-customer obligations imposed on operators of computing clusters.
- Price transparency and anti-discrimination provisions for models and clusters.
- Enforcement and whistleblower protections.
- Establishment of a Frontier Model Division within CDT.
- Creation of CalCompute.
The central focus of this bill is ensuring that developers of “covered models,” essentially extremely powerful AI systems, are proceeding with caution given the enormous potential for harm posed by them in the hands of malicious actors. At the core of these obligations is the “limited duty exemption,” which is a determination that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
It is important to note the magnitude of the computing power necessary to meet the definition of covered model and, likewise, the magnitude of the hazardous capabilities that the bill seeks to prevent.
First, “covered models” are artificial intelligence models trained using a quantity of computing power greater than 10^26 integer of floating-point operations. Floating-point operations, or FLOP, is a measure of the amount of compute used in AI systems. This threshold of FLOP is currently out of reach for all but a handful of entities. Therefore the bill focuses on the most highly capable AI models, often referred to as “frontier models.” Some of today’s examples of frontier models are GPT-4 (OpenAI), Claude 3 (Anthropic), and Gemini Ultra (Google).
The author explains this metric:
SB 1047 uses this threshold to have developers begin testing for hazardous capabilities because it indicates that the model is larger than any model trained today, and without testing it is impossible to rule out the chance that the next generation of models will contain hazardous capabilities.
The average performance of AI models is predictably related to how much computational power, measured in FLOP, was used to train them. Performance on any individual task will tend to go up with FLOP used to train, but with less predictability. The largest models trained so far are believed to have used only about 10^25 FLOP. Safety testing from AI developers and independent technical auditors suggests that, while current models do not yet possess hazardous capabilities, they display warning signs indicating that substantially larger models may possess such capabilities. As computational training power moves beyond 10^26 FLOP, therefore, it becomes prudent and reasonable to test for these capabilities before deploying them.
The goal of this is ensure the models powerful enough to cause critical harms continue to be covered by the bill as metrics for determining the appropriate thresholds advance. Writing in opposition, a coalition of industry associations led by the California Chamber of Commerce raise issues with the definition:
There is little to no certainty as to what this translates to in practice and, in any case, such thresholds will become obsolete within a year, requiring the law to change yet again. Moreover, by equating model size to risk, the definition of “covered models” is simultaneously overly broad and too narrow as smaller and/or less performant models can present much greater risks than large/higher performant ones. As a result, the bill both fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones.
With regard to the magnitude of harms being targeted, the level of potential carnage required to constitute a “hazardous capability” is substantial. It must be capable of being used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
- The creation or use of a chemical, biological, radiological, or nuclear (CBRN) weapon in a manner that results in mass casualties.
- At least $500 million of damage through cyberattacks on critical infrastructure.
- At least $500 million of damage by a model that autonomously engages in conduct that would be criminal if done by a human.
The definition also encompasses other comparably severe threats to public safety and security and those capabilities made possible by fine tuning and post-training modifications performed by third-party experts intending to demonstrate those abilities.
A detailed safety and security protocol must be put into place that includes specification of testing to be done and describes in detail how the developer will meet the various security requirements of the bill. The bill requires developers to ensure that these protocol are properly implemented as written, including by designating specific senior personnel responsible for implementation and conducting audits, as appropriate.
The bill also requires the developer to follow all “covered guidance,” which is defined to include:
- Guidance issued by NIST and by the Frontier Model Division that is relevant to the management of safety risks associated with AI models that may possess hazardous capabilities.
- Industry best practices, including safety practices, precautions, or testing procedures undertaken by developers of comparable models that are relevant to the management of safety risks associated with AI models that may possess hazardous capabilities.
The bill does not prohibit initiating such widespread use of a model where a limited duty exemption has not been made. Rather it requires the developer to implement reasonable safeguards and other security measures to prevent an individual from being able to use the hazardous capabilities of the model to cause a critical harm or to use the model to create a derivative model to cause a critical harm. The developer is required to provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm. Developers must ensure, to the extent reasonably possible, that the covered model’s actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
However, the developer is prohibited from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
For those models still not subject to a limited duty exemption, developers are required to annually certify, under the penalty of perjury, compliance with the safety and security requirements of the bill to Frontier Models Division.
The Frontier Model Division. This bill establishes the Frontier Model Division, within CDT, and requires the division, among other things, to review annual certification reports, advise the AG on potential violations, issue guidance, standards, and best practices, publish anonymized AI safety incident reports, and establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices.
Additionally, this bill requires the Frontier Model Division to appoint and consult with an advisory committee to advise the Governor on when it may be necessary to proclaim a state of emergency relating to AI and advise the Governor what responses may be appropriate in that event. Existing law, California’s Emergency Services Act (ESA) gives the Governor the authority to proclaim a state of emergency in an area affected or likely to be affected when: a) conditions of disaster or extreme peril exist; b) the Governor is requested to do so upon request from a designated local government official; or c) the Governor finds that local authority is inadequate to cope with the emergency. Local governments may also issue local emergency proclamations, which is a prerequisite for requesting the Governor’s Proclamation of a State of Emergency.
Specifically, the ESA authorizes the Governor to proclaim a state of emergency when conditions of disaster or of extreme peril to the safety of persons and property within the state caused by conditions such as cyberterrorism, sudden and severe energy shortage, and electromagnetic pulse attack, or other conditions which, by reason of their magnitude, are or are likely to be beyond the control of the services, personnel, equipment, and facilities of any single county or city occur.
Further, this bill authorizes the Frontier Model Division to monitor relevant developments relating to the safety risks associated with the development of AI models and the functioning of markets for AI models and, on or before July 1, 2026, issue guidance regarding technical thresholds and benchmarks relevant to determining whether an AI model is a covered model and technical thresholds and benchmarks relevant to determining whether a covered model is subject to a limited duty exemption.
CalCompute. This bill directs CDT to commission consultants to create a public cloud computing cluster, to be known as CalCompute, with the primary focus of conducting research into the safe and secure deployment of large-scale AI models and fostering equitable innovation. Specifically, the consultants are required to be, but not limited to, representatives of national laboratories, universities, and any relevant professional associations or private sector stakeholders.
To meet the objective of establishing CalCompute, this bill requires CDT to ensure that the consultants commissioned to work on this process evaluate and incorporate all of the following considerations:
- An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, as specified.
- The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
- A framework to determine the parameters for use of CalCompute, as specified.
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
- An analysis of the state’s investment in the training and development of the technology workforce, including through degree programs at the UC, the CSU, and the CCCs.
- A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
A cloud computing cluster is a group of connected servers that work together as a single system, often over the internet, to perform computing tasks. This setup allows for the pooling of resources, such as processing power and storage, from multiple machines. This collective resource pool can be used more efficiently and flexibly than single-server or single-computer solutions. Back in the day, the machines used for high performance computing were known as “supercomputers,” or big standalone machines with specialized hardware – very different from what you find in home and office computers.
Nowadays, however, the majority of supercomputers are instead computer clusters. These inter-connected computers are endowed with software to coordinate programs on (or across) those computers, and they can therefore work together to perform computationally intensive tasks.
Distributing computing tasks across several servers can significantly enhance performance. Large computational tasks can be divided into smaller parts and executed simultaneously across multiple machines, drastically reducing processing time. Further, cloud clusters often utilize virtualization technologies, which allow for the efficient use of hardware, reducing the cost associated with physical hardware investments and maintenance.
There are currently several existing cloud computing clusters primarily operated by major technology companies and organizations. These clusters serve a variety of purposes from general cloud computing needs to more specialized applications like AI research and big data analytics. A few prominent examples include: Google’s Cloud Platform offering cloud computing services that include powerful compute clusters; Amazon Web Services providing a comprehensive and widely used suite of cloud computing services for large-scale computing tasks, including AI model training and deployment; Microsoft Azure which supports large-scale computing tasks similar to those intended for CalCompute; and the European Organization for Nuclear Research (CERN) openlab, while not a commercial service provider, CERN’s cloud computing resources are a notable example of public cloud capabilities for scientific research. They provide a powerful infrastructure for processing vast amounts of scientific data, particularly from particle physics experiments.
As proposed by this bill, CalCompute is intended to represent a significant step towards integrating advanced cloud computing capabilities with public sector innovation and research in AI. The establishment of CalCompute attempts to harness the growing power of AI technologies while addressing critical aspects such as safety, security, and equitable access and leans on the involvement of a diverse range of stakeholders, from national labs to local universities and the private sector.
This bill requires CDT to submit an annual report to the Legislature from the commissioned consultants on CalCompute to ensure progress in meeting the objectives laid out in the bill, and authorizes CDT to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate CalCompute. This bill provides that CalCompute shall only become operative upon an appropriation in a budget act, as specified.
The provisions of this bill are severable, that is, if any provision of the Safe and Secure Innovation for Frontier AI Systems Act or its application are held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.
Prior/Related Legislation
SB 313 (Dodd, 2023) would have established the Office of AI, as specified, and require state agencies to disclose when they are using generative AI (GenAI) to communicate with a person and to provide them an option to speak with a natural person at the agency. (Held on the Senate Appropriations Committee Suspense File)
SB 398 (Wahab, 2023) would have established the AI for California Research Act, which requires CDT to develop and implement a comprehensive research plan to study the feasibility of using advanced technology to improve state and local government services. (Never heard in the Senate Committee on Governmental Organization)
SB 892 (Padilla, 2024) requires CDT to establish safety, privacy, and nondiscrimination standards relating to artificial intelligence services, as defined. Commencing August 1, 2025, the bill prohibits a contract for AI services, as defined, from being entered into by the state unless the provider meets those standards. CDT is required to report to the Legislature regarding the standards it establishes. (Pending in the Senate Committee on Appropriations)
SB 893 (Padilla, 2024) requires GovOps, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California AI Research Hub in GovOps, as specified. (Pending in the Senate Committee on Appropriations)
SB 896 (Dodd, 2024) the AI Accountability Act, establishes guidelines for the use and regulation of AI and (GenAI) within state government operations. Specifically, this bill requires the development and production of a Benefits and Risk of Generative Artificial Intelligence Report; requires joint risk analyses of potential threats posed by GenAI to California’s critical energy infrastructure, as specified; requires the development of guidelines for the procurement of AI and GenAI, as specified; requires the support and training of the state government workforce for the next generation of skills needed to thrive in the GenAI economy, as specified; and, requires a state agency to identify a person when that person is communicating with a state agency directly through GenAI, as specified. (Pending in the Senate Committee on Appropriations)
SB 942 (Becker, 2024) the AI Transparency Act, requires specified providers of GenAI to create an AI detection tool available to the public to determine the extent to which text, image, video, audio, or multimedia content was created by a GenAI system, as specified; requires those providers to include a visible disclosure in AI-generated content including a clear and conspicuous notice that identifies the content as generated by AI; and creates the Generative AI Registry Fund and requires those providers to register with CDT, as specified. (Pending in the Senate Committee on Governmental Organization)
SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.
AB 302 (Ward, Chapter 800, Statutes of 2023) requires CDT, in coordination with other interagency bodies, to conduct a comprehensive inventory of all high-risk automated decision systems used by state agencies on or before September 1, 2024, and report the findings to the Legislature by January 1, 2025, and annually thereafter, as specified.
AB 331 (Bauer-Kahan, 2023) would have prohibited “algorithmic discrimination,” that is, use of an automated decision tool to contribute to unjustified differential treatment or outcomes that may have a significant effect on a person’s life, as specified. (Held on the Senate Appropriations Committee Suspense File)
AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. (Pending in the Assembly Committee on Privacy and Consumer Protection)
AB 2885 (Bauer-Kahan, 2024) defines AI and inserts references to the provided definition throughout existing code, as specified. (Pending in the Assembly Committee on Appropriations)
AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer and a developer of an automated decision tool to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The assessments must be provided to the Civil Rights Department within 7 days of a request. (Pending in the Assembly Committee on Judiciary)
SJR 6 (Chang, Res. Chapter 112, Statutes of 2019) urged the President and the Congress of the United States to develop a comprehensive AI Advisory Committee and to adopt a comprehensive AI policy.
ACR 215 (Kiley, Res. Chapter 206, Statutes of 2018) expressed the Legislature’s support for a set of principles for the governance of AI known as the 23 Asilomar AI Principles.
SB 532 (Dodd, Chapter 557, Statutes of 2017) added “cyberterrorism” to the list of conditions constituting a state of emergency and a local emergency.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes
SUPPORT:
Center for AI Safety Action Fund (Co-source)
Economic Security California Action (Co-source)
Encode Justice (Co-source)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program
Denizen
Depict.AI
District Council of Iron Workers for the State of California
Elicit
ENH Alpha LLC
FAR AI, Inc.
Fathom Radiant
Foresight Institute
ForHumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA: StateStrong
KIRA Center for AI Risks & Impacts
Latino Community Foundation
Lionheart Ventures
Los Angeles Area Chamber of Commerce
Loveable Labs Incorporated
MIT AI Alignment
ML Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group
OPPOSITION:
Association of National Advertisers
California Chamber of Commerce
California Land title Association
California Manufacturers and Technology Association
Chamber of Progress
Civil Justice Association of California
Computer and Communications Industry Association
Context Fund
Insights Association
Silicon Valley Leadership Group
Software and Information Industry Association
TechNet
ARGUMENTS IN SUPPORT: A coalition of supporters write that, “SB 1047 requires developers to conduct self-assessments and take precautions to prevent three major risks: cyberattacks causing over $500 million in damage, AI-enabled crimes resulting in damages exceeding $500 million, and the creation of novel weapons of mass destruction using AI. Developers must implement safety measures before training and deploying risky models, monitor and report incidents, and comply with safety protocols. The Attorney General of California is empowered to take enforcement action against developers whose negligence poses imminent threats to public safety or causes severe harm to Californian citizens.”
Further, “SB 1047 also aims to foster AI innovation and support smaller startups and researchers. It establishes a public cloud-computing cluster for researching safe and secure AI deployment, allowing diverse stakeholders to participate in the development of large-scale AI systems. The bill creates an advisory council to advocate for safe and secure open-source AI development and mandates transparent pricing and non-discriminatory practices from cloud-computing companies and frontier model developers to ensure equal opportunities for smaller players in the AI ecosystem.”
ARGUMENTS IN OPPOSITION: Opponents of the bill write that, “SB 1047 seeks to regulate frontier AI developers from innovating AI models that will result in any kind of foreseeable harm—even harms that would not manifest from the model itself. In doing so, the bill requires developers to comply with incredibly vague, broad, impractical, if not impossible, standards when developing ‘covered models’. For example, SB 1047 applies to AI models that either: (1) meet a size threshold (a computing power greater than 10^26 integer or floating-point operations in 2024), or (2) that perform similarly. What the latter category of covered models looks like, however, is not entirely clear. (See Proposed Section 22602(f).) The bill merely states that they are models ‘trained using a quantity of computing power sufficiently large that it could be reasonably expected to have similar or greater performance as an AI model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models’. There is little to no certainty as to what this translates to in practice and, in any case, such thresholds will become obsolete within a year, requiring the law to change yet again. Moreover, by equating model size to risk, the definition of ‘covered models’ is simultaneously overly broad and too narrow as smaller and/or less performant models can present much greater risks than large/higher performant ones. As a result, the bill both fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones.”
Further, “[w]e cannot overemphasize the importance of ensuring consistency in the AI regulatory landscape, nationally, and the need to follow federal guidance on certain issues that transcend national borders. Relevant to this bill, in October 2023, the White House issued an Executive Order (EO) that requires companies that are developing any foundation model that poses a serious risk to national security, national economic security, or national public health and safety to notify the federal government when training the model and share the results of all red-team safety tests to ensure that AI systems are safe, security and trustworthy before companies make them public. While we appreciate that in some respects, SB 1047 appears in line with the goals of the federal government and the White House’s EO, the [NIST] is already working with other agencies at the federal level to establish testing and safety guidelines for large models. If enacted, SB 1047 would likely result in confusion about the correct standards to apply and place additional burdens on AI developers without commensurate gains in safety, especially as it fails to align with regulations nationally and introduces novel concepts and standards including around the assessment of what is a ‘hazardous capability’. Indeed, given the definition of ‘covered models’ under this bill which also scopes in any fine-tuning by downstream customers and users, SB 1047 is more far-reaching than anything seen to date in voluntary commitments, federal guidance, or the laws of any other countries.”
DUAL REFERRAL: Senate Judiciary Committee (9-0) & Senate Governmental Organization Committee
Senator Thomas Umberg, Chair
2023-2024 Regular Session
Bill No: SB 1047 (Author:Wiener)
Version: March 20, 2024
Hearing Date: April 2, 2024
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
Safe and Secure Innovation for Frontier Artificial Intelligence Systems Act
DIGEST
This bill requires developers of powerful artificial intelligence models and those providing the computing power to train such models to put appropriate safeguards and policies into place to prevent critical harms. The bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computing cluster.
EXECUTIVE SUMMARY
Owing to recent advances in processing power and the rise of big data, artificial intelligence’s (AI) capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact business, and consume media. It has been used to accelerate productivity, achieve efficiencies, liberate us from drudgery, write our college essay, help us understand and enjoy the world, upgrade the Pope’s fashion, connect with each other, and live longer, fuller lives. It has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy. For good or ill, its transformative potential seems boundless.
This bill seeks to establish guardrails for the most powerful AI models to avoid the more catastrophic possibilities about which experts have raised alarms. It places a series of obligations on developers of “covered models” and providers of the cloud compute for training such models. The bill also seeks to establish a public cloud-computing cluster that facilitates equitable participation in the development and deployment of responsible AI systems. The bill is co-sponsored by the Center for AI Safety Action Fund, Economic Security California Action, and Encode Justice. It is supported by a host of tech companies and labor organizations and opposed by the Chamber of Progress and a coalition of industry associations.
PROPOSED CHANGES TO THE LAW
Existing law:
- Establishes the California Department of Technology (CDT) within the Government Operations Agency, under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Gov. Code Sec. 11545(a).)
- Provides that the duties of the Director include:
- advising the Governor on the strategic management and direction of the state’s information technology (IT) resources;
- establishing and enforcing state IT strategic plans, policies, standards, and enterprise architecture, as specified;
- minimizing overlap, redundancy, and cost in state IT operations by promoting the efficient and effective use of information technology;
- providing technology direction to agency and department chief information officers to ensure the integration of statewide technology initiatives, compliance with IT policies and standards, and the promotion of the alignment and effective management of IT services;
- working to improve organizational maturity and capacity in the effective management of IT; and establishing performance management and improvement processes to ensure state IT systems and services are efficient and effective. (Gov. Code § 11545(b).)
- Provides that persons are responsible, not only for the result of their willful acts, but also for an injury occasioned to another by their want of ordinary care or skill in the management of their property or person, except so far as the latter has, willfully or by want of ordinary care, brought the injury upon themselves. (Civ. Code § 1714(a).)
This bill:
- Establishes the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
- Provides definitions for all relevant terms, including:
- “Artificial intelligence model” means a machine-based system designed to operate with varying levels of autonomy that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments.
- “Covered model” means an artificial intelligence model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations (FLOP) or a model that can reasonably be expected to have similar performance capabilities as assessed by commonly used benchmarks.[1]
- “Positive safety determination” means a determination with respect to a covered model that a developer can reasonably exclude the possibility that the model has a “hazardous capability” or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
- “Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
- The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.
- At least $500 million of damage through cyberattacks on critical infrastructure.
- At least $500 million of damage by a model that autonomously engages in conduct that would be criminal if done by a human.
- Other comparably severe threats to public safety and security.
- “Hazardous capability” includes the capabilities above even if they would not manifest but for fine tuning and posttraining modifications performed by third-party experts.
- “Fine tuning” means the adjustment of the model weights of an artificial intelligence model after it has finished its initial training by training the model with new data.
- “Covered guidance” means any of the following:
- Applicable guidance issued by the National Institute of Standards and Technology and by the Frontier Model Division.
- Industry best practices, including relevant safety practices, precautions, or testing procedures undertaken by developers of comparable models, and any safety standards or best practices commonly or generally recognized by relevant experts in academia or the nonprofit sector.
- Applicable safety-enhancing standards set by standards setting organizations.
- “Computing cluster” means a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
- Establishes the Frontier Model Division (FMD) within CDT and tasks them with various duties, including the following:
- Review annual certification reports from developers.
- Advise the Attorney General on potential violations of this law.
- Issue guidance, standards, and best practices sufficient to prevent unreasonable risks from covered models with hazardous capabilities.
- Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
- Establish an accreditation process and relevant accreditation standards under which third parties may be accredited to certify adherence by developers to the best practices and standards adopted.
- Publish anonymized artificial intelligence safety incident reports received from developers.
- Develop and submit to the Judicial Council proposed model jury instructions for actions brought by individuals injured by a hazardous capability of a covered model.
- Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
- Appoint and consult with various advisory committees.
- Requires developers of “covered models” to determine whether they can make a “positive safety determination” with respect to the model before initiating training. The developer is required to incorporate all “covered guidance.” The determination shall be certified under penalty of perjury and submitted to the FMD, specifying the basis for it.
- Requires developers of covered models unable to make such a safety determination, before training the model and until a positive safety determination can be made, to implement various safeguards. This includes:
- Implement administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, that are appropriate given the associated risks.
- Implement the capability to enact a full shutdown or a “kill switch.”
- Implement all covered guidance.
- Implement, and annually review, a written and separate safety and security protocol, as specified, that details how it is adequate to prevent critical harms and “hazardous capabilities” and specifies the testing procedures incorporated therein. The protocol shall be provided to FMD.
- Ensures that these protocol are implemented as written, including by designating senior personnel responsible and conducting audits, as appropriate.
- Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm.
- Requires a developer of a covered model without a positive safety determination, upon completion of training, to perform capability testing sufficient to determine whether a positive safety determination can be made, pursuant to its safety and security protocol.
- Provides, that if a positive safety determination is then made, the developer shall submit to FMD a certification of compliance with the basis and specific methodology and results. This shall be done within 90 days and no more than 30 days after initiating the commercial, public, or widespread use of the covered model.
- Provides that a developer that makes a good faith error regarding a positive safety determination shall be deemed to be in compliance if it reports its error to FMD within 30 days of completing the training and ceases operation of the model until the developer is otherwise in compliance. However, reliance on an unreasonable positive safety determination, as defined, does not relieve a developer of these obligations.
- Requires a developer, if a positive safety determination is not made, before initiating the commercial, public, or widespread use of the covered model to implement reasonable safeguards and requirements to prevent an individual from being able to use the hazardous capabilities of the model, or a derivative model, to cause a critical harm or use the model to create a derivative model that is used to cause a critical harm. Reasonable requirements must be placed on developers of derivative models to further ensure prevention of these harms. The developer must ensure, to the extent reasonably possible, that the covered model’s actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible.
- Requires a developer to refrain from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
- Establishes a continuing responsibility on developers to periodically reevaluate the procedures, policies, protections, capabilities, and safeguards implemented in light of the growing capabilities of covered models and as is reasonably necessary to ensure that the covered model or its users cannot remove or bypass them. For models still not subject to a positive safety determination, developers must annually certify their compliance with these provisions to FMD. The certification must be signed by the chief technology officer, or a more senior corporate officer, as prescribed by FMD, and include details of the risks the model may pose. Safety incidents must be reported to FMD, as prescribed, and without unreasonable delay, but in no event more than 72 hours later.
- Prohibits a developer from preventing an employee from disclosing information to the Attorney General if the employee has reasonable cause to believe that the information indicates that the developer is out of compliance. A developer shall not retaliate against an employee for disclosing such information. Developers must provide clear notice to all employees working on covered models of their rights and responsibilities under this section. The Attorney General may publicly release any complaint, or a summary of that complaint, if disclosure will serve the public interest.
- Requires persons that operate computing clusters to implement appropriate written policies and procedures to do the following when a customer utilizes compute resources that would be sufficient to train a covered model:
- Obtain, and annually validate, a prospective customer’s basic identifying information and business purpose for utilizing the computing cluster, as specified.
- Annually assess whether a prospective customer intends to utilize the cluster to deploy a covered model.
- Maintain for seven years and provide to the Frontier Model Division or the Attorney General, records of actions taken pursuant to this law.
- Implement the ability to promptly enact a full shutdown in the event of an emergency.
- Requires a developer of a covered model that provides commercial access to it to provide a transparent, uniform, publicly available price schedule for the purchase of access to that model at a given level of quality and quantity subject to the developer’s terms of service and prohibits developers from engaging in unlawful discrimination or noncompetitive activity in determining price or access. Operators of computing clusters are required to do the same with respect to computing clusters. However, a person that operates a computing cluster may provide free, discounted, or preferential access to public entities, academic institutions, or for noncommercial research purposes.
- Authorizes the Attorney General, if they have reasonable cause to believe that a person is in violation of these provisions, to bring an action seeking recovery of preventive relief, including a permanent or temporary injunction, restraining order, or other order against the person responsible for the violation, including deletion of the covered model and the weights utilized in that model. Monetary damages to persons aggrieved and a court order for a full shutdown are also available. However, these remedies are only available in response to harm or an imminent risk or threat to public safety. The Attorney General may also recover a civil penalty in an amount not exceeding 10 percent of the cost, excluding labor, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor, to develop the covered model for any subsequent violation.
- Subjects liable defendants to joint and several liability and instructs the court to disregard corporate formalities under specific conditions:
- Where steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
- Where the corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief.
- Clarifies that the duties and obligations imposed are cumulative with any other duties or obligations imposed under other law and shall not be construed to relieve any party from any duties or obligations imposed under other law and do not limit any rights or remedies under existing law.
- Tasks CDT with creating a public cloud computing cluster known as CalCompute through the commissioning of consultants with specified objectives, first of which is to study the safe and secure deployment of large-scale AI models. The consultants shall include representatives of national laboratories, public universities, and any relevant professional associations or private sector stakeholders. They shall evaluate and incorporate the following considerations into their plan:
- An analysis of the public, private, and nonprofit cloud platform infrastructure ecosystem, including, but not limited to, dominant cloud providers, the relative compute power of each provider, the estimated cost of supporting platforms as well as pricing models, and recommendations on the scope of CalCompute.
- The process to establish affiliate and other partnership relationships to establish and maintain an advanced computing infrastructure.
- A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- An evaluation of the landscape of existing computing capability, resources, data, and human expertise in California for the purposes of responding quickly to a security, health, or natural disaster emergency.
- An analysis of the state’s investment in the training and development of the technology workforce, including through degree programs at the University of California, the California State University, and the California Community Colleges.
- A process for evaluating the potential impact of CalCompute on retaining technology professionals in the public workforce.
- Authorizes CDT to receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate the establishment of CalCompute.
COMMENTS
- Defining AI
Reminiscent of the wave of legislation to regulate social media platforms, there are dozens of bills currently making their way through the Legislature to regulate AI. Before potentially subjecting the industry and regulators to a host of new laws, it seems imperative to first define exactly what it is we are talking about and to harmonize the definitions being used just as was done when the first comprehensive definition of “social media platform” was codified a few years ago.
The task is not necessarily a straightforward one, as pointed out by the United States Congressional Research Service:
Defining AI is not merely an academic exercise, particularly when drafting legislation. AI research and applications are evolving rapidly. Thus, congressional consideration of whether to include a definition for AI in a bill, and if so how to define the term or related terms, necessarily include attention to the scope of the legislation and the current and future applicability of the definition. Considerations in crafting a definition for use in legislation include whether it is expansive enough not to hinder the future applicability of a law as AI develops and evolves, while being narrow enough to provide clarity on the entities the law affects. Some stakeholders, recognizing the many challenges of defining AI, have attempted to define principles that might help guide policymakers. Research suggests that differences in definitions used to identify AI-related research may contribute to significantly different analyses and outcomes regarding AI competition, investments, technology transfer, and application forecasts.[2]
There are several leading definitions of AI that can be relied on to craft a reasonably clear yet appropriately broad definition for this advancing technology. While there is no definition of AI in state law, the California Privacy Protection Agency (CPPA) has published draft regulations that provide a definition for AI:
“Artificial intelligence” means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The artificial intelligence may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different artificial intelligence varies in its levels of autonomy and adaptiveness after deployment. For example, artificial intelligence includes generative models, such as large language models, that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.[3]
At the federal level, the National Artificial Intelligence Act of 2020 provides the following:
Artificial intelligence. The term “artificial intelligence” means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments. Artificial intelligence systems use machine and human-based inputs to—
(A) perceive real and virtual environments;
(B) abstract such perceptions into models through analysis in an automated manner; and
(C) use model inference to formulate options for information or action.[4]
Also at the federal level, the National Institute for Standards and Technology (NIST), in their Artificial Intelligence Risk Management Framework (AI RMF 1.0), defined an AI system as “an engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.”
At the international level, the Organization for Economic Co-operation and Development (OECD) published and then revised a definition for AI to serve as a standard across jurisdictions:
An AI system is a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. Different AI systems vary in their levels of autonomy and adaptiveness after deployment.[5]
With the revision, OECD made the case for harmonization:
Obtaining consensus on a definition for an AI system in any sector or group of experts has proven to be a complicated task. However, if governments are to legislate and regulate AI, they need a definition to act as a foundation. Given the global nature of AI, if all governments can agree on the same definition, it allows for interoperability across jurisdictions.[6]
Most recently, the European Parliament signed the EU AI Act, which defines AI as follows:
“AI system” means a machine-based system designed to operate with varying levels of autonomy, that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.
In order to gain both the benefit of the expertise and compromise that went into formulating these definitions and the efficiencies that come with harmonization, the Committee, with a variety of stakeholders, including the author, have come up with the following definition to begin this process and to amend into the bill, which incorporates elements of the NIST definition into the internationally recognized formulations while eliminating unnecessary examples from within it:
“Artificial intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
- Frameworks for responsible development and accountability in AI
With recent dramatic advances in the capabilities of AI systems, the need for frameworks for accountability and responsible development have become ever more urgent.
In January of 2017, AI researchers, economists, legal scholars, ethicists, and philosophers met in Asilomar, California to discuss principles for managing the responsible development of AI. The collaboration resulted in the Asilomar Principles. Aspirational rather than prescriptive, these 23 principles were intended to initiate and frame a dialogue by providing direction and guidance for policymakers, researchers, and developers. Its endorsers include 1,200 leading experts in the field of AI, including DeepMind founder Demis Hassabis and the late Stephen Hawking.
The Legislature subsequently adopted ACR 215 (Kiley, Ch. 206, Stats. 2018), which added the State of California to that list by endorsing the Asilomar Principles as guiding values for the development of artificial intelligence and related public policy. In broad strokes, those principles aim to do the following:
- Research issues: create beneficial AI; direct funding toward beneficial innovation; maintain constructive and healthy exchanges between AI researchers and policymakers; promote a culture of trust, cooperation, and transparency among researchers and developers of AI; and avoid corner-cutting on safety standards.
- Ethics and values: promote safety, failure transparency, judicial transparency, and responsible innovation; align human values with innovation; protect privacy and liberty; ensure that the benefits and prosperity created by AI are broadly shared; maintain human control over AI; develop AI that supports rather than subverts social and civil processes; and avoid an AI arms race.
- Longer-term issues: avoid assumptions regarding the capabilities of AI; give AI its due attention; and recognize that its risks are potentially catastrophic or existential. [emphasis added]
As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.
While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.
More recently the Biden Administration has published its Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the
design, use, and deployment of AI to protect the rights of the American public:
- Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
- Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
- Data Privacy: You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used. You should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used. Systems should not employ user experience and design decisions that obfuscate user choice or burden users with defaults that are privacy invasive. Consent should only be used to justify collection of data in cases where it can be appropriately and meaningfully given. Any consent requests should be brief, be understandable in plain language, and give you agency over data collection and the specific context of use; current hard-to-understand notice-and-choice practices for broad uses of data should be changed. Enhanced protections and restrictions for data and inferences related to sensitive domains, including health, work, education, criminal justice, and finance, and for data pertaining to youth should put you first. In sensitive domains, your data and related inferences should only be used for necessary functions, and you should be protected by ethical review and use prohibitions. You and your communities should be free from unchecked surveillance; surveillance technologies should be subject to heightened oversight that includes at least pre-deployment assessment of their potential harms and scope limits to protect privacy and civil liberties. Continuous surveillance and monitoring should not be used in education, work, housing, or in other contexts where the use of such surveillance technologies is likely to limit rights, opportunities, or access. Whenever possible, you should have access to reporting that confirms your data decisions have been respected and provides an assessment of the potential impact of surveillance technologies on your rights, opportunities, or access.
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.[7]
TechEquity, an organization committed to ensuring technology’s evolution benefits everyone equitably, has also laid out their straightforward AI Policy Principles:
- People who are impacted by AI must have agency to shape the technology that dictates their access to critical needs like employment, housing, and healthcare.
- The burden of proof must lie with developers, vendors, and deployers to demonstrate that their tools do not create harm—and regulators, as well as private [individuals], should be empowered to hold them accountable.
- Concentrated power and information asymmetries must be addressed in order to effectively regulate the technology.
The need for thoughtful regulation and accountability is especially urgent with regard to the existential risks that many believe unfettered AI advancement poses. In response to these risks, the Future of Life Institute published an open letter early last year, calling for a pause on giant AI experiments:
Contemporary AI systems are now becoming human-competitive at general tasks, and we must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders. Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable. This confidence must be well justified and increase with the magnitude of a system's potential effects. OpenAI's recent statement regarding artificial general intelligence, states that "At some point, it may be important to get independent review before starting to train future systems, and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models." We agree. That point is now.
Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium.[8]
Signatories to the letter include Stuart Russell, Berkeley, Professor of Computer Science, director of the Center for Intelligent Systems, and co-author of the standard textbook “Artificial Intelligence: a Modern Approach"; Elon Musk, CEO of SpaceX, Tesla & X; and Steve Wozniak, Co-founder, Apple.
Subsequent to that letter, the Center for AI Safety released another open letter signed by a wide-ranging group of industry leaders, researchers, and engineers working in AI that highlighted the existential risk posed by unethical AI development and the urgency of the issue. The statement simply read: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”[9]
This was signed by the most cited researchers of AI, including Dr. Yoshua Bengio and Dr. Geoffrey Hinton; both Turing Award winners and considered the “godfathers” of modern AI. In addition, prominent executives at the leading AI development companies also signed on, including Ilya Sutskever, co-founder and chief scientist, OpenAI; Sam Altman, chief executive of OpenAI; Demis Hassabis, chief executive of Google DeepMind; and Dario Amodei, chief executive of Anthropic.
While the future is unclear, the need to respond to these potential harms now is evident. The Center for New American Security puts a fine point on it:
While there is significant uncertainty in how the future of AI develops, current trends point to a future of vastly more powerful AI systems than today’s state of the art. The most advanced systems at AI’s frontier will be limited initially to a small number of actors but may rapidly proliferate. Policymakers should begin to put in place today a regulatory framework to prepare for this future. Building an anticipatory regulatory framework is essential because of the disconnect in speeds between AI progress and the policymaking process, the difficulty in predicting the capabilities of new AI systems for specific tasks, and the speed with which AI models proliferate today, absent regulation. Waiting to regulate frontier AI systems until concrete harms materialize will almost certainly result in regulation being too late.[10]
- Ensuring the safe and secure development of AI
This bill takes a number of approaches in responding to concerns in AI regulation. The bill can generally be broken up into the following elements:
- Safety and security obligations imposed on developers of powerful AI models.
- Know-your-customer obligations imposed on operators of computing clusters.
- Price transparency and anti-discrimination provisions for models and clusters.
- Enforcement and whistleblower protections.
- Establishment of a Frontier Model Division within CDT.
- Creation of CalCompute.
- Guardrails in connection with the development and deployment of large AI models
The central focus of this bill is ensuring that developers of “covered models,” essentially extremely powerful AI systems, are proceeding with caution given the enormous potential for harm posed by them in the hands of malicious actors. At the core of these obligations is the “positive safety determination,” which is a determination that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.
It is important to note the magnitude of the computing power necessary to meet the definition of covered model and, likewise, the magnitude of the hazardous capabilities that the bill seeks to prevent.
First, “covered models” are artificial intelligence models trained using a quantity of computing power greater than 10^26 integer of floating-point operations. Floating-point operations, or FLOP, is a measure of the amount of compute used in AI systems. This threshold of FLOP is currently out of reach for all but a handful of entities. Therefore the bill focuses on the most highly capable AI models, often referred to as “frontier models.” Some of today’s examples of frontier models are GPT-4 (OpenAI), Claude 3 (Anthropic), and Gemini Ultra (Google).
The author explains this metric:
SB 1047 uses this threshold to have developers begin testing for hazardous capabilities because it indicates that the model is larger than any model trained today, and without testing it is impossible to rule out the chance that the next generation of models will contain hazardous capabilities.
The average performance of AI models is predictably related to how much computational power, measured in FLOP, was used to train them. Performance on any individual task will tend to go up with FLOP used to train, but with less predictability. The largest models trained so far are believed to have used only about 10^25 FLOP. Safety testing from AI developers and independent technical auditors suggests that, while current models do not yet possess hazardous capabilities, they display warning signs indicating that substantially larger models may possess such capabilities. As computational training power moves beyond 10^26 FLOP, therefore, it becomes prudent and reasonable to test for these capabilities before deploying them.
President Biden’s executive order on AI included heightened requirements that companies developing large foundation models report on a regular basis to the federal government. This bill uses the same threshold used in connection with those requirements.
Given the rapid development and advancement of these models and the efficiency of their training, concerns have been raised that this metric will be insufficient before long. Recognizing this reality, the bill alternatively provides the following threshold for a “covered model”:
The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.
The goal of this is ensure the models powerful enough to cause critical harms continue to be covered by the bill as metrics for determining the appropriate thresholds advance. Writing in opposition, a coalition of industry associations led by the California Chamber of Commerce raise issues with the definition:
There is little to no certainty as to what this translates to in practice and, in any case, such thresholds will become obsolete within a year, requiring the law to change yet again. Moreover, by equating model size to risk, the definition of “covered models” is simultaneously overly broad and too narrow as smaller and/or less performant models can present much greater risks than large/higher performant ones. As a result, the bill both fails to adequately address the very real risks posed by small but malicious models and imposes significant costs on innovating performant but responsible ones.
While understanding the difficulty of creating defined parameters for ever-evolving AI systems, this definition suffers from a lack of clarity. For instance, who determines what the “benchmarks commonly used to quantify the general performance of state-of-the-art foundation models” are? The author has committed to working to increase clarity around how one can determine what models are included in the future.
Second, with regard to the magnitude of harms being targeted, the level of potential carnage required to constitute a “hazardous capability” is substantial. It must be capable of being used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:
- The creation or use of a chemical, biological, radiological, or nuclear (CBRN) weapon in a manner that results in mass casualties.
- At least $500 million of damage through cyberattacks on critical infrastructure.
- At least $500 million of damage by a model that autonomously engages in conduct that would be criminal if done by a human.
The definition also encompasses other comparably severe threats to public safety and security and those capabilities made possible by fine tuning and posttraining modifications performed by third-party experts intending to demonstrate those abilities.
Writing in support, the Future of Life Institute recommends some changes to ensure a more inclusive universe of critical harms:
Though facilitating the creation or use of a chemical, biological, radiological, or nuclear (CBRN) weapon “in a manner that results in mass casualties” is considered a hazardous capability for a covered system, facilitating damage through cyberattacks on critical infrastructure or autonomous criminal conduct only rises to the level of a “hazardous capability” if it results in at least $500 million of damage. (Bus. & Prof. Code Sec. 22602(n)) This $500 million dollar threshold for damage is remarkably high, and would be extremely challenging to practically assess in preliminary testing. A system capable of, e.g., facilitating cyberattacks on critical infrastructure, should be properly assessed with appropriate steps taken to mitigate this risk, regardless of whether that potential damage would verifiably exceed $500 million. Furthermore, while the definition of hazardous capability in the context of cyberattacks and criminal conduct includes damage to property, CBRN risk only qualifies if it results in mass casualties. We recommend that the author remove the $500 million dollar threshold for qualifying as a hazardous capability in the cases of cyberattacks and criminal conduct, and include damage to property as a qualifying risk of facilitating CBRN development.
The author argues that given the unknown capabilities of future frontier models, reasonable guardrails should be in place for the largest of them. Therefore, the bill requires developers of such models to determine whether it can make a “positive safety determination” before initiating training of the model, essentially a finding that their model is not capable of being used to cause these large-scale harms. The developer would then certify, under penalty of perjury, such a determination to a division within CDT that shall be established by the bill, and discussed in more detail below, the Frontier Model Division (FMD).
The coalition in opposition argues against this approach:
SB 1047 still makes it impossible for developers to actually make any positive safety determinations ruling out those hazardous capabilities by requiring developers to make the positive safety determination before they initiate training of the covered model. (Proposed Section 22603). Because a developer needs to test the model by training it in a controlled environment to make a positive safety determination and yet cannot train a model until such a determination is made, SB 1047 effectively places developers in a perpetual catch-22 and illogically prevents them from training frontier models altogether.
However, the bill does not require developers to make a positive safety determination before they initiate training. The bill requires a developer to “determine whether it can make a positive safety determination” before initiating training. In fact, it is likely that given the power and seemingly boundless potential of these models that most developers will be unable to make a positive safety determination before training it.
The bill does not require the cessation of development and training should such a determination not be made. Rather, the bill requires developers to implement protections and protocol before initiating training of the models if they determine that a positive safety determination cannot be reached. This includes implementation of administrative, technical, and physical cybersecurity protections to prevent unauthorized access to, or misuse or unsafe modification of, the covered model, including to prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights from the developer’s custody. Developers must also ensure there is a “kill switch,” essentially the ability to cease operation of a covered model, including all copies and derivative models, on all computers and storage devices.
A detailed safety and security protocol must be put into place that includes specification of testing to be done and describes in detail how the developer will meet the various security requirements of the bill. The bill requires developers to ensure that these protocol are properly implemented as written, including by designating specific senior personnel responsible for implementation and conducting audits, as appropriate.
The bill also requires the developer to follow all “covered guidance,” which is defined to include:
- Applicable guidance issued by the National Institute of Standards and Technology (NIST) and FMD.
- Industry best practices, including relevant safety practices, precautions, or testing procedures undertaken by developers of comparable models, and any safety standards or best practices commonly or generally recognized by relevant experts in academia or the nonprofit sector.
- Applicable safety-enhancing standards set by standards setting organizations.
This requirement is imposed before training, both when determining whether a positive safety determination can be made, and if not, thereafter. The goal is to ensure that AI developers incorporate safety practices that may be generally acknowledged as necessary for managing hazardous capabilities of covered models. However, the coalition in opposition argues the language is unworkable. They write:
SB 1047 also requires that a developer “incorporate all covered guidance” before making a positive safety determination. However, industry and others are still trying to ascertain how to define what constitutes a high-capable, foundational model and it is unclear what will qualify as “industry best practices” or “standards setting organizations” for the purpose of incorporating all covered guidance. Such regulatory uncertainty will inevitably discourage economic and technological innovation. It would make far more sense to let the U.S. AI Safety Institute to complete its work first, after which safety and security protocols tied to those safety standards could be considered.
The open-ended definition of covered guidance is intentional as the author points out that it is not yet possible to specify all appropriate guidelines in full detail, or the specific organizations that will provide them, as the field will be adapting over time. However, greater clarity around the obligations imposed on developers may be necessary as the current language is arguably overly vague. For instance, while reasonableness is a common standard in tort law, its appearance in so many of the definitions and standards in the bill arguably causes concern regarding the interpretability of the bill. Accordingly, the author has committed to continuing to work on finding the right balance on this front. The goal is to have widely accepted safety and security standards put into place and to ensure they are implemented properly. The language in the bill must ensure there are concrete benchmarks and standards both to ensure developers have clear direction but also for regulators and public prosecutors to have the ability to effectively hold them to those standards.
After completing training of a covered model that is not subject to a positive safety determination, developers are again required to perform specified testing pursuant to its safety and security protocol to attempt to make such a determination. If such a determination is made, the developer must submit certification to FMD within 90 days but no later than 30 days after initiating the commercial, public, or widespread use of the covered model. The author may wish to harmonize the certification process at all points in the development process with that provided for positive safety determinations made at the pre-training phase.
The bill does not prohibit initiating such widespread use of a model where a positive safety determination has not been made. Rather it requires the developer to implement reasonable safeguards and other security measures to prevent an individual from being able to use the hazardous capabilities of the model to cause a critical harm or to use the model to create a derivative model to cause a critical harm. The developer is required to provide reasonable requirements to developers of derivative models to prevent an individual from being able to use a derivative model to cause a critical harm. Developers must ensure, to the extent reasonably possible, that the covered model’s actions and any resulting critical harms can be accurately and reliably attributed to it and any user responsible for those actions.
However, the developer is prohibited from initiating the commercial, public, or widespread use of a covered model if there remains an unreasonable risk that an individual may be able to use the hazardous capabilities of the model, or a derivative model based on it, to cause a critical harm.
For those models still not subject to a positive safety determination, developers are required to annually certify compliance with the safety and security requirements of the bill to FMD.
Again, these provisions are generally aligned with the focus in President Biden’s executive order with respect to the development of models of this size. The EO orders the Secretary of Commerce to require developers of these models to report, on an ongoing basis, to the federal government regarding the training and development of these models and the cybersecurity protections taken; the ownership and possession of the model weights; and the results of performance testing. In fact, many companies have already voluntarily committed to follow specified guidelines that track with the bill’s security and safety obligations. As described in the White House fact sheet:
President Biden [convened] seven leading AI companies at the White House [] – Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI – to announce that the Biden-Harris Administration has secured voluntary commitments from these companies to help move toward safe, secure, and transparent development of AI technology.
Companies that are developing these emerging technologies have a responsibility to ensure their products are safe. To make the most of AI’s potential, the Biden-Harris Administration is encouraging this industry to uphold the highest standards to ensure that innovation doesn’t come at the expense of Americans’ rights and safety.
These commitments, which the companies have chosen to undertake immediately, underscore three principles that must be fundamental to the future of AI – safety, security, and trust – and mark a critical step toward developing responsible AI. As the pace of innovation continues to accelerate, the Biden-Harris Administration will continue to remind these companies of their responsibilities and take decisive action to keep Americans safe.[11]
Some of the relevant commitments are:
Ensuring Products are Safe Before Introducing Them to the Public
- The companies commit to internal and external security testing of their AI systems before their release. This testing, which will be carried out in part by independent experts, guards against some of the most significant sources of AI risks, such as biosecurity and cybersecurity, as well as its broader societal effects.
- The companies commit to sharing information across the industry and with governments, civil society, and academia on managing AI risks. This includes best practices for safety, information on attempts to circumvent safeguards, and technical collaboration.
Building Systems that Put Security First
- The companies commit to investing in cybersecurity and insider threat safeguards to protect proprietary and unreleased model weights. These model weights are the most essential part of an AI system, and the companies agree that it is vital that the model weights be released only when intended and when security risks are considered.
- The companies commit to facilitating third-party discovery and reporting of vulnerabilities in their AI systems. Some issues may persist even after an AI system is released and a robust reporting mechanism enables them to be found and fixed quickly.
Therefore, the very companies that would be regulated by this bill committed to implementing similar safeguards. In testimony before the United States Senate Committee on the Judiciary’s Subcommittee on Privacy, Technology, and the Law, Sam Altman, CEO of OpenAI made clear how important such safeguards are, stressing that “it is vital that AI companies–especially those working on the most powerful models–adhere to an appropriate set of safety requirements, including internal and external testing prior to release and publication of evaluation results.”[12] He detailed some of the rigorous measures undertaken before initiating widespread use of such models:
Prior to releasing each new version of our models, OpenAI conducts extensive testing, engages external experts for feedback, improves the model's behavior with techniques like reinforcement learning from human feedback (RLHF), and implements safety and monitoring systems.
The release of our latest model, GPT-4, provides an illustrative example. After we developed GPT-4, we spent more than 6 months evaluating, testing, and improving the system before making it publicly available.
In addition to our own evaluations, we engaged with external AI safety experts in a process known as “red teaming,” through which they helped identify potential concerns with GPT-4 in areas including the generation of inaccurate information (known as “hallucinations”), hateful content, disinformation, and information related to the proliferation of conventional and unconventional weapons.
This process helped us to better understand potential usage risks and ways to address those risks. In each of these areas, we developed mitigations to increase safety in significant ways. Some of our work involved making adjustments to the data used to train the model, during what is called the pre-training stage. Other interventions took place after initial training of the model.
However, opposition argues that the certification requirements in this bill are “unaccountably broad and hopelessly unworkable.” Chamber of Progress writes:
Unfortunately, SB 1047 forces model developers to engage in speculative fiction about imagined threats of machines run amok, computer models spun out of control, and other nightmare scenarios for which there is no basis in reality.
Instead, SB 1047 forces developers operating in the real world to proactively mitigate against every conceivable harm - and many inconceivable ones - not just by the model itself, but subsequent third parties who make use of the model.
As stated, the author has committed to continuing to work to hone the specific requirements laid out in the bill. The author states the need for these provisions:
With the political prospects for joint action between the White House and Congress in doubt, California has an indispensable role to play in ensuring that we develop this extremely powerful technology with basic guardrails, in order to allow society to experience its massive potential benefits. Clarifying that developers of the largest and most powerful AI models must take basic precautions against serious risks to public safety and national security is the clear way forward. Establishing these duties, in line with industry-leading best practices, will not address all of AI’s risks and harms, but it is a necessary step forward. And no state is better positioned to take on this challenge than California.
The concerns about workability and clarity are important to engage with, and honing the precise protocols that should be put in place are critical as this bill moves through the legislative process. Establishing the exact protocols and policies that should be put into place for such a rapidly-advancing technology is inherently difficult. However, the author may wish to create more clarity in the defined terms and the specified guidance that should be followed. Just as important, the author may wish to bolster the role of FMD and the Attorney General, and possibly other entities, to ensure there is proper oversight of this development and to confirm that the appropriate and necessary safeguards are being put into place, perhaps through authorizing or mandating government or accredited third-party auditing. Again the Future of Life Institute provides suggestions for more robust oversight:
Before initiating training of a covered model, the bill in print requires a developer to “ensure that the safety and security protocol is implemented as written” and recommends doing so by, among other things, “conducting, audits, including through third parties as appropriate.” (Bus. & Prof. Code Sec. 22603(b)(5)) Third party auditing is essential to ensuring that ulterior motives do not compromise the integrity of safety protocol assessment, especially when it comes to systems potentially capable of critical harm. Put simply, Big Tech companies should not be grading their own homework with respect to reasonable safety protocols. To ensure independent assessment and accountability to the spirit of the bill, we recommend that the author consider amending the bill to: 1) authorize the Frontier Model Division to audit compliance with the bill’s requirements as appropriate, or 2) require independent audits by third parties certified by the Frontier Model Division to perform them to the standards laid out by the bill and other guidance.
Given the level of complexity and the technological sophistication that will be necessary to adequately oversee this development, reliance on such independent audits may be critical as well as a focus on building up the resources of the FMD.
It should be noted that Chamber of Progress’ assertions that the threats of critical harm from advanced AI systems are “imagined” and have “no basis in reality” stands in direct contrast to the very clear public proclamations from industry leaders and leading academics in the world of AI. As discussed more thoroughly above, “mitigating the risk of extinction from AI should be a global priority.” Geoffrey Hinton, the godfather of AI referenced above, says about the bill:
Forty years ago when I was training the first version of the AI algorithms behind tools like ChatGPT, no one - including myself - would have predicted how far AI would progress. Powerful AI systems bring incredible promise, but the risks are also very real and should be taken extremely seriously.
SB 1047 takes a very sensible approach to balance those concerns. I am still passionate about the potential for AI to save lives through improvements in science and medicine, but it’s critical that we have legislation with real teeth to address the risks. California is a natural place for that to start, as it is the place this technology has taken off.
The other godfather, Dr. Bengio, shares this perspective:
AI systems beyond a certain level of capability can pose meaningful risks to democracies and public safety. Therefore, they should be properly tested and subject to appropriate safety measures. This bill offers a practical approach to accomplishing this, and is a major step toward the requirements that I've recommended to legislators.
- Operators of computing clusters: know your customers
The bill also imposes security and safety obligations on those providing the cloud compute to train these larger models. “Computing cluster” is defined as a set of machines transitively connected by data center networking of over 100 gigabits per second that has a theoretical maximum computing capacity of at least 10^20 integer or floating-point operations per second and can be used for training artificial intelligence.
The bill requires a person that operates a computing cluster to obtain a prospective customer’s basic identifying and contact information and the business purpose for utilizing the computing cluster. This includes the means and source of payment, including any associated financial institution, credit card number, account number, customer identifier, transaction identifiers, or virtual currency wallet or wallet address identifier. The operator of the computing cluster must also obtain the Internet Protocol addresses used for access or administration and the date and time of each access or administrative action.
The operator must also, at least annually, assess whether the customer intends to use the computing cluster to deploy a covered model and to validate the above information. Records of actions taken and policies put into place to satisfy these provisions must be maintained for seven years and provided to FMD or the Attorney General upon request. Operators must also implement the capability to promptly enact a full shutdown in the event of an emergency.
All of these duties are triggered only when a customer utilizes enough compute resources that would be sufficient to train the massive models covered by the bill. The author explains that this section of the bill is requiring “companies that provide cloud compute for frontier model training to institute ‘know your customer’ policies to help prevent the dangerous misuse of AI systems by malicious actors and geopolitical adversaries.”
- Enforcement
The Attorney General is authorized to bring a civil action against those reasonably believed to be in violation of these provisions. The Attorney General may seek preventative relief ranging from a temporary injunction to deletion of the covered model and the weights utilized. However, any such relief is only available in response to harm or an imminent risk or threat to public safety.
Courts are authorized to impose civil penalties in an amount not exceeding 10 percent of the cost, excluding labor cost, to develop the covered model for a first violation and in an amount not exceeding 30 percent of the cost, excluding labor cost, to develop the covered model for any subsequent violation. In the apportionment of penalties assessed pursuant to this section, defendants shall be jointly and severally liable.
Anticipating potential corporate shell games, the bill provides that the court shall disregard corporate formalities and impose joint and several liability on affiliated entities for purposes of effectuating the intent of the bill if the court concludes that both of the following are true:
- Steps were taken in the development of the corporate structure among affiliated entities to purposely and unreasonably limit or avoid liability.
- The corporate structure of the developer or affiliated entities would frustrate recovery of penalties or injunctive relief under this section.
To ensure violations are brought to light by those closest to the actual development of these models, the bill provides whistleblower protections prohibiting developers from preventing employees from disclosing information about compliance issues to the Attorney General or retaliating against employees when they do so.
Notwithstanding the above, the bill provides that a developer that makes a good faith error regarding such a determination is still considered in compliance so long as the developer reports its error to FMD within 30 days of completing the training of the covered model and ceases operation of it until the developer is otherwise in compliance. However, in another section, the bill states that reliance on an unreasonable positive safety determination does not relieve a developer of its obligations under this section. A positive safety determination is unreasonable if the developer does not take into account reasonably foreseeable risks of harm or weaknesses in capability testing that lead to an inaccurate determination. They are reasonably foreseeable, if, by the time that a developer releases a model, an applicable risk of harm or weakness in capability testing has already been identified by another developer of a comparable model or
the United States Artificial Intelligence Safety Institute, the FMD, or any independent standard-setting organization or capability-testing organization cited by either of those entities.
- Establishment of the FMD
To serve as the central governmental entity overseeing the development of frontier AI models in California, the bill establishes the Frontier Model Division within CDT. As seen from the discussion above, the FMD will receive and maintain the reports, certifications, policies and protocol, and other information required to be submitted by developers. The bill also lays out a series of other duties, including:
- Advise the Attorney General on potential violations.
- Issue guidance, standards, and best practices sufficient to prevent unreasonable risks from covered models with hazardous capabilities including, but not limited to, more specific requirements on the duties required of developers.
- Establish an accreditation process and relevant accreditation standards under which third parties may be accredited to certify adherence by developers to the best practices and standards.
- Publish anonymized artificial intelligence safety incident reports received from developers pursuant to Section 22603 of the Business and Professions Code.
- Establish confidential fora that are structured and facilitated in a manner that allows developers to share best risk management practices for models with hazardous capabilities in a manner consistent with state and federal antitrust laws.
- Appoint and consult with an advisory committee that shall advise the Governor on when it may be necessary to proclaim a state of emergency relating to artificial intelligence and advise the Governor on what responses may be appropriate in that event.
- Appoint and consult with an advisory committee for open-source artificial intelligence that, among other things, will advise FMD on future policies and legislation impacting open-source artificial intelligence development.
- Provide technical assistance and advice to the Legislature, upon request, with respect to artificial intelligence-related legislation.
- Develop and submit to the Judicial Council proposed model jury instructions for actions brought by individuals injured by a hazardous capability of a covered model.
- Responding to market dynamics
As industry races toward developing larger, more powerful AI models and seeks to commodify the seemingly infinite applications of AI, concerns are growing about the diminishing role that researchers, academic institutions, and more public-focused entities are playing in the development of AI. As reported by the Washington Post:
As such tech behemoths as Meta, Google and Microsoft funnel billions of dollars into AI, a massive resources gap is building with even the country’s richest universities. Meta aims to procure 350,000 of the specialized computer chips — called GPUs — that are essential to run the gargantuan calculations needed for AI models. In contrast, Stanford’s Natural Language Processing Group has 68 GPUs for all of its work.
To obtain the expensive computing power and data required to research AI systems, scholars frequently partner with tech employees. Meanwhile, tech firms’ eye-popping salaries are draining academia of star talent.
Big tech companies now dominate breakthroughs in the field. In 2022, the tech industry created 32 significant machine learning models, while academics produced three, a significant reversal from 2014, when the majority of AI breakthroughs originated in universities, according to a Stanford report.
Researchers say this lopsided power dynamic is shaping the field in subtle ways, pushing AI scholars to tailor their research for commercial use. Last month, Meta CEO Mark Zuckerberg announced that the company’s independent AI research lab would move closer to its product team, ensuring “some level of alignment” between the groups, he said.
“The public sector is now significantly lagging in resources and talent compared to that of industry,” said [Fei-Fei] Li, a former Google employee and the co-director of the Stanford Institute for Human-Centered AI. “This will have profound consequences because industry is focused on developing technology that is profit-driven, whereas public-sector AI goals are focused on creating public goods.”
…
As Silicon Valley races to build chatbots and image generators, it is drawing would-be computer science professors with high salaries and the chance to work on interesting AI problems. Nearly 70 percent of people with PhDs in AI end up in private industry compared with 21 percent of graduates two decades ago, according to a 2023 report.[13]
The bill seeks to address this by first creating “CalCompute,” a public cloud computing cluster created by CDT. The focus of the project will be conducting research into the safe and secure deployment of large-scale artificial intelligence models and fostering equitable innovation that includes:
- A fully owned and hosted cloud platform.
- Necessary human expertise to operate and maintain the platform.
- Necessary human expertise to support, train, and facilitate use of CalCompute.
CDT is specifically directed to commission an array of consultants and direct them to incorporate specified considerations into their plan. Two important considerations are:
- A process for evaluating appropriate uses of the public cloud resources and their potential downstream impact, including mitigating downstream harms in deployment.
- A framework to determine the parameters for use of CalCompute, including, but not limited to, a process for deciding which projects will be supported by CalCompute and what resources and services will be provided to projects.
CDT is required to report annually to the Legislature. Fealty to these, and the other stated, considerations is critical for the project to meet its lofty goals. The author may wish to further fortify, including through clear transparency and accountability measures, this aspect of the program to ensure that these resources and capabilities are truly working toward the public benefit and avoid cooption by the larger industry. The Future of Life Institute provides similar recommendation:
Gov. Code Sec. 11547.7(e) of the bill in print authorizes the California Department of Technology to “receive private donations, grants, and local funds, in addition to allocated funding in the annual budget, to effectuate” the creation of its public cloud computing infrastructure. We recognize that budgetary constraints may limit the availability of public funds to realize the promising vision of this project, but we caution that dependence on private funds, which will likely come from the very Big Tech companies developing risky AI systems subject to the bill’s requirements, can implicitly facilitate regulatory capture. Without proper constraints, this ambitious and impactful state project, if reliant on private donations for its subsistence, could succumb to influence from the profit-driven interests of industry leaders with the most disposable resources, and associated regulatory efforts in the future may similarly be beholden to these interests. We recommend that the author consider clearly defining limitations on the identification and input of private donors in this process, and explicitly prioritize the use of public funds over private funds where possible.
The Chamber of Progress, writing in opposition to the bill, commends the inclusion of CalCompute.
In addition to creating this public resource, the bill addresses the issues raised above by requiring developers of covered models that provide commercial access to it to provide a transparent, uniform, publicly available price schedule for the purchase of access to that covered model at a given level of quality and quantity subject to the developer’s terms of service and prohibits them from engaging in unlawful discrimination or noncompetitive activity in determining price or access. The same obligations are imposed on operators of computing clusters with regard to access to their computing clusters. The author explains this section of the bill: “[I]n order to ensure that smaller startup developers have equal opportunities to larger players, SB 1047 requires cloud-computing companies and frontier model developers to provide transparent pricing and avoid price discrimination.”
Economic Security California Action, a co-sponsor of the bill, explains the need for these provisions:
The AI sector is seemingly very dynamic, with weekly headlines detailing major players buying and selling, investing in and spinning out different enterprises. However, all that movement belies the fact that AI is inarguably among America’s most concentrated markets, with three large companies – Amazon, Google, and Microsoft – controlling access to two of the most expensive resources needed for AI: cloud computing power and large models. These same companies also own both upstream and downstream business interests, allowing for preferential pricing or deals that benefit some competitors over others. As a result, small entities and start-ups – those not owned by the big players – are at a significant disadvantage. To foster a competitive and innovative AI sector, SB 1047 requires cloud-computing companies and frontier model developers to provide fair and transparent pricing and disallows price discrimination.
- Stakeholder positions
According to the author:
Large-scale artificial intelligence has the potential to produce an incredible range of benefits for Californians and our economy—from advances in medicine and climate science to improved wildfire forecasting and clean power development. It also gives us an opportunity to apply hard lessons learned over the last decade, as we’ve seen the consequences of allowing the unchecked growth of new technology without evaluating, understanding, or mitigating the risks. SB 1047 does just that, by developing responsible, appropriate guardrails around development of the largest, most powerful AI systems, to ensure they are used to improve Californians’ lives, without compromising safety or security.
SB 1047 will also promote the growth of the AI industry by establishing CalCompute, a public AI research cluster that will allow startups, researchers, and community groups to participate in the development of large-scale AI systems. By providing a broad range of stakeholders with access to the AI development process, CalCompute will help align large-scale AI systems with the values and needs of California communities.
The Center for AI Safety, a co-sponsor of the bill, explains the need for legislative action:
Universities, startups, and technology companies across California are harnessing AI to enhance scientific research, spur new economic opportunities, and boost human creativity. With continued responsible development, AI has vast potential to improve Californians' lives in areas from healthcare to sustainability.
However, leading AI researchers have warned that failure to take appropriate precautions could have severe consequences for public safety and security as developers produce increasingly powerful AI models. In a 2023 letter organized by the Center for AI Safety, hundreds of AI experts, academics, and industry leaders stated that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” . . .
Just within this past week, SEC Chair Gary Gensler warned that unchecked AI systems could lead to a future financial meltdown, a group of eminent scientists in the US and China stressed that countries must set "red lines" around particular kinds of risky AI development, and a new spate of impressive demonstrations were released showcasing powerful AI agents that are capable of writing code and taking hundreds of independent actions without human oversight. There are warning signs that the next generation of models may have novel dangerous capabilities. Independent researchers at METR (a Berkeley-based non-profit formerly known as ARC Evals), who performed early safety testing on some of OpenAI and Anthropic’s latest model release[s], has explicitly said that “for systems more capable than Claude and GPT-4, we are now at the point where we need to check carefully that new models do not have sufficient capabilities to replicate autonomously or cause catastrophic harm – it’s no longer obvious that they won’t be able to.”
Encode Justice, another co-sponsor of the bill, writes:
SB 1047 introduces essential safeguards for the creation of highly capable AI models, often known as “frontier AI models.” These models are defined in the bill as trained using over 10^26 floating-point operations. Models of this scope would cost at least $100 million to develop and, notably, do not yet publicly exist but are anticipated to emerge soon as technological advancements continue.
These are advanced, resource-intensive projects that have caught attention at the highest levels of government and are the focus of President Biden’s Executive Order on Artificial Intelligence for their significant national security and public safety implications.
SB 1047’s Key Safeguards for Frontier AI:
- Pre-Deployment: Developers need to rigorously test these AI giants for any risks or unexpected behaviors.
- Cybersecurity: To block hackers and prevent misuse, tight security measures are a must to protect these models from being compromised.
- Limiting Harmful Uses: If testing reveals any potentially dangerous abilities within these models, developers must restrict these capabilities to prevent misuse.
- Ongoing Monitoring: After these models go live, their behavior must be continuously monitored. Developers need to be ready to address new risks as they emerge, including shutting down the model if necessary.
SB 1047 focuses on the developers of the most advanced models, promoting best practices already in use by leading companies. This law doesn’t burden smaller AI startups or the broader business and academic community using AI tools.
Chamber of Progress argues in opposition:
It is critical that public policy foster an abundance of frontier models - open and closed alike, existing and new entrants. A plurality of models will catalyze AI application development and ultimately benefit consumers. However, SB 1047 gives the largest incumbent AI models and models built upon them (“derivative models”) special treatment that will inevitably lead to fewer upstart (“non-derivative”) models. This will entrench the largest incumbent players in AI frontier model development - making them even more consequential - and undercut innovation when we should be encouraging a proliferation of approaches.
The disparate treatment is unaccountable and is not rationalized in the legislative text. Indeed it cannot be rationalized. Any regulation should apply fairly across approaches. SB 1047 would have the effect of freezing model innovation in its place.
Developers build applications on top of models; competition at the model level will mean increased innovation at a lower cost. This in turn will promote the equitable diffusion of AI’s benefits across California.
In response to these and other concerns regarding the bill’s impact on innovation, the author states:
SB 1047’s requirements only apply to a very small handful of frontier model AI developers that are training the largest, most capital intensive models - today costing in excess of $100 million dollars. The vast majority of AI startups, and all AI application and use-case developers, would have zero new duties under SB 1047. And SB 1047’s duties are modeled on responsible best practices pioneered by leading developers in the industry itself.
The coalition of industry associations, including the Association of National Advertisers, writes in opposition:
We cannot overemphasize the importance of ensuring consistency in the AI regulatory landscape, nationally, and the need to follow federal guidance on certain issues that transcend national borders. Relevant to this bill, in October 2023, the White House issued an Executive Order (EO) that requires companies that are developing any foundation model that poses a serious risk to national security, national economic security, or national public health and safety to notify the federal government when training the model and share the results of all red-team safety tests to ensure that AI systems are safe, secur[e] and trustworthy before companies make them public.
While we appreciate that in some respects, SB 1047 appears in line with the goals of the federal government and the White House’s EO, the National Institute of Standards and Technology (NIST) is already working with other agencies at the federal level to establish testing and safety guidelines for large models. If enacted, SB 1047 would likely result in confusion about the correct standards to apply and place additional burdens on AI developers without commensurate gains in safety, especially as it fails to align with regulations nationally and introduces novel concepts and standards including around the assessment of what is a “hazardous capability”. Indeed, given the definition of “covered models” under this bill which also scopes in any fine-tuning by downstream customers and users, SB 1047 is more far-reaching than anything seen to date in voluntary commitments, federal guidance, or the laws of any other countries.
Ultimately, enacting a patchwork of inconsistent AI regulations that go into as much detail as SB 1047, will further fracture the U.S. regulatory landscape.
SUPPORT
Center for AI Safety Action Fund (sponsor)
Economic Security Project Action (sponsor)
Encode Justice (sponsor)
AE Studio
AI Safety Student Team (Harvard)
Apart Research
Cambridge Boston Alignment Initiative
Causative Labs
Civic AI Security Program
Denizen
Depict.ai
District Council of Iron Workers of the State of California and Vicinity
Elicit
Enh Alpha LLC
Far AI, INC.
Fathom Radiant
Foresight Institute
Forhumanity
Future of Life Institute
General Agents
General Proximity
Gladstone AI
Higher Ground Labs
Imbue
Indivisible CA Statestrong
Kira Center for Ai Risks & Impacts
Lionheart Ventures
Loveable Labs Incorporated
MIT AI Alignment
Ml Alignment & Theory Scholars
Momentum
Mythos Ventures
New Media Studio
Nonlinear
Normative
Panoplia Laboratories
Paper Farms
Redwood Research
Safe AI Future
The Future Society
White Space Marketing Group
OPPOSITION
Association of National Advertisers
California Chamber of Commerce
California Manufacturers and Technology Association
Chamber of Progress
Civil Justice Association of California
Computer and Communications Industry Association
Insights Association
Software and Information Industry Association
Technet
RELATED LEGISLATION
Pending Legislation:
SB 313 (Dodd, 2023) establishes the Office of Artificial Intelligence. It requires state agencies to disclose when they are using generative artificial intelligence to communicate with a person and to provide them an option to speak with a natural person at the agency. SB 313 was held on suspense in the Senate Appropriations Committee.
SB 398 (Wahab, 2023) establishes the Artificial Intelligence for California Research Act, which requires CDT to develop and implement a comprehensive research plan to study the feasibility of using advanced technology to improve state and local government services. SB 398 was never heard in the Senate Governmental Organization Committee.
SB 892 (Padilla, 2024) requires CDT to establish safety, privacy, and nondiscrimination standards relating to artificial intelligence services, as defined. Commencing August 1, 2025, the bill prohibits a contract for AI services, as defined, from being entered into by the state unless the provider meets those standards. CDT is required to report to the Legislature regarding the standards it establishes. SB 892 is currently in the Senate Governmental Organization Committee.
SB 893 (Padilla, 2024) requires the Government Operations Agency, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California Artificial Intelligence Research Hub in the Government Operations Agency, as prescribed. SB 893 requires the hub to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development that seeks to harness the technology’s full potential for public benefit while safeguarding privacy, advancing security, and addressing risks and potential harms to society, as prescribed. SB 893 is currently in the Senate Governmental Organization Committee.
SB 896 (Dodd, 2024) requires the Government Operations Agency, the Department of Technology, and the Office of Data and Innovation to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, as prescribed. The bill would require, as often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities to perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
The bill also requires a state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state agency or department is being communicated through artificial intelligence. The also requires an automated decisionmaking system, as defined, utilized by a state agency or department to be evaluated for risk potential before adoption, as specified. SB 896 is currently in the Senate Governmental Organization Committee.
SB 942 (Becker, 2024) establishes the California AI Transparency Act, which, among other things, requires a covered provider, as defined, to create an AI detection tool by which a person can query the covered provider as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a generative AI system, as defined, provided by the covered provider that meets certain criteria. Covered providers are required to include in AI-generated content a visible disclosure that, among other things, includes a clear and conspicuous notice, that identifies the content as generated by AI. SB 942 requires a covered provider to register with CDT and provide them a URL to any AI detection tool it has created. SB 942 is currently in this Committee.
SCR 17 (Dodd, 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems. SCR 17 is currently in the Assembly Privacy and Consumer Protection Committee.
AB 331 (Bauer-Kahan, 2023) prohibits “algorithmic discrimination,” that is, use of an automated decision tool to contribute to unjustified differential treatment or outcomes that may have a significant effect on a person’s life. It requires any deployer of an automated decision tool to perform an impact assessment for those tools and to notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision. AB 331 was held in the Senate Appropriations Committee.
AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. AB 2013 is currently in the Assembly Privacy and Consumer Protection Committee.
AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer and a developer of an automated decision tool to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The assessments must be provided to the Civil Rights Department within 7 days of a request. AB 2930 requires a deployer to, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool.
AB 2930 is currently in the Assembly Privacy and Consumer Protection Committee.
Prior Legislation:
AB 302 (Ward, Ch. 800, Stats. 2023) requires the California Department of Technology (CDT), in coordination with other interagency bodies, to conduct a comprehensive inventory of all high-risk automated decision systems (ADS) used by state agencies on or before September 1, 2024, and report the findings to the Legislature by January 1, 2025, and annually thereafter, as specified.
AB 13 (Chau, 2021) would have established the Automated Decision Systems Accountability Act, which, in the context of the State’s procurement policies, promotes oversight over ADS that pose a high risk of adverse impacts on individual rights. The bill was eventually gutted and amended to address a different topic.
SB 444 (Umberg, 2019) would have requested the Regents of the University of California (UC) to enact a resolution authorizing the law schools at UC Berkeley and UC Irvine to participate in a pilot project to develop AI or machine-learning solutions to address access to justice issues faced by self-representing litigants in their respective courts. The bill died in the Assembly Higher Education Committee.
AB 1576 (Calderon, 2019) would have required the Secretary of Government Operations to appoint participants to an AI working group to evaluate the uses, risks, benefits, and legal implications associated with the development and deployment of AI by California-based businesses. The bill was held on the Senate Appropriations Committee suspense file.
SJR 6 (Chang, Res. Ch. 112, Stats. 2019) urged the President and the Congress of the United States to develop a comprehensive AI Advisory Committee and to adopt a comprehensive AI policy.
ACR 215 (Kiley, Resolution Ch. 206, Stats. 2018) expressed the Legislature’s support for a set of principles for the governance of AI known as the 23 Asilomar AI Principles.
**************
In the course of drafting previous amendments, language in the definition of covered model remained that was intended to be taken out. The author has agreed to amendments that would remove “in 2024” from Section 22602(f)(1). ↑
Laurie A. Harris, Artificial Intelligence: Background, Selected Issues, and Policy Considerations (May 19, 2021) Congressional Research Service, https://crsreports.congress.gov/product/pdf/R/R46795. All internet citations are current as of March 26, 2024. ↑
Draft Risk Assessment and Automated Decisionmaking Technology Regulations (March 2024) CPPA, https://cppa.ca.gov/meetings/materials/20240308_item4_draft_risk.pdf. ↑
15 U.S.C. § 9401. This definition was subsequently cited in President Biden’s executive order on AI. ↑
Recommendation of the Council on Artificial Intelligence (July 11, 2023) OECD, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449. ↑
Stuart Russell, Karine Perset & Marko Grobelnik, Updates to the OECD’s definition of an AI system explained (November 29, 2023) OECD, https://oecd.ai/en/wonk/ai-system-definition-update. ↑
Blueprint For An AI Bill Of Rights (October 2022) Office of Science and Technology Policy, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. ↑
Future of Life Institute, Pause Giant AI Experiments: An Open Letter (2023) https://futureoflife.org/open-letter/pause-giant-ai-experiments/ [emphasis in original]. ↑
Statement on AI Risk (2023) Center for AI Safety, https://www.safe.ai/work/statement-on-ai-risk. ↑
Paul Scharre, Future-Proofing Frontier AI Regulation (March 2024) Center for New American Security, https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS-Report_AI-Trends_FinalC.pdf. ↑
FACT SHEET: Biden-Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI (July 21, 2023) The White House, https://www.whitehouse.gov/briefing-room/statements-releases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-to-manage-the-risks-posed-by-ai/. ↑
Sam Altman, Written Testimony before the U.S. Senate Committee on the Judiciary, https://www.judiciary.senate.gov/imo/media/doc/2023-05-16%20-%20Bio%20&%20Testimony%20-%20Altman.pdf. ↑
Naomi Nix, Cat Zakrzewski & Gerrit De Vynck, Silicon Valley is pricing academics out of AI research (March 10, 2024) The Washington Post, https://www.washingtonpost.com/technology/2024/03/10/big-tech-companies-ai-research/. ↑
Discussed in Hearing
Senate Floor
Assembly Floor
Assembly Standing Committee on Judiciary
Assembly Standing Committee on Privacy and Consumer Protection
Senate Floor
Senate Standing Committee on Governmental Organization
Senate Standing Committee on Governmental Organization
Senate Standing Committee on Governmental Organization
Bill Author
Bill Co-Author(s):