SB 896: Generative Artificial Intelligence Accountability Act.
- Session Year: 2023-2024
- House: Senate
- Latest Version Date: 2024-09-29
Current Status:
Passed
(2024-09-29: Chaptered by Secretary of State. Chapter 928, Statutes of 2024.)
Introduced
In Committee
First Chamber
In Committee
Second Chamber
Enacted
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Generative Artificial Intelligence Accountability Act, would, among other things, require the Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the Department of Human Resources, to update the report to the Governor, as required by Executive Order No. N-12-23, as prescribed. The bill would require the Office of Emergency Services to, as appropriate, perform a risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical infrastructure, including those that could lead to mass casualty events and would require a high-level summary of the analysis be provided annually to the Legislature.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person regarding government services and benefits to ensure that those communications include both (1) a disclaimer that indicates to the person that the communication was generated by generative artificial intelligence and (2) information, as specified, describing how the person may contact a human employee of the state agency or department.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Generative Artificial Intelligence Accountability Act, would, among other things, require the Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the Department of Human Resources, to update the report to the Governor, as required by Executive Order No. N-12-23, as prescribed. The bill would require the Office of Emergency Services to, as appropriate, perform a risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical infrastructure, including those that could lead to mass casualty events and would require a high-level summary of the analysis be provided annually to the Legislature.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person regarding government services and benefits to ensure that those communications include both (1) a disclaimer that indicates to the person that the communication was generated by generative artificial intelligence and (2) information, as specified, describing how the person may contact a human employee of the state agency or department.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Generative Artificial Intelligence Accountability Act, would, among other things, require the Government Operations Agency, the Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the California Privacy Protection Agency Department of Human Resources, to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, report to the Governor, as required by Executive Order No. N-12-23, as prescribed. The bill would require the Office of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center every 2 years to Services to, as appropriate, perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. events and would require a high-level summary of the analysis be provided annually to the Legislature.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person regarding government services and benefits to ensure that those communications include both (1) a disclaimer that indicates to the person that the communication was generated by generative artificial intelligence and (2) clear instructions, information, as specified, describing how the person may contact a human employee of the state agency or department.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Generative Artificial Intelligence Accountability Act, would, among other things, require the Government Operations Agency, the Department of Technology, and the Office of Data and Innovation Innovation, and the California Privacy Protection Agency to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, as prescribed. The bill would require, as often as is deemed appropriate by the Director require the Office of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities Center every 2 years to perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify person to ensure that those communications include both (1) a disclaimer that indicates to that the person that the persons interaction with communication was generated by generative artificial intelligence and (2) clear instructions, as specified, describing how the person may contact a human employee of the state agency or department is being communicated through artificial intelligence. department.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Generative Artificial Intelligence Accountability Act, would, among other things, require the Government Operations Agency, the Department of Technology, and the Office of Data and Innovation to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, as prescribed. The bill would require, as often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities to perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Artificial Intelligence Accountability Act, would, among other things, require the Government Operations Agency, the Department of Technology, and the Office of Data and Innovation to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, as prescribed. The bill would require, as often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities to perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
This bill would also require an automated decisionmaking system, as defined, utilized by a state agency or department to be evaluated for risk potential before adoption, as specified.Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Artificial Intelligence Accountability Act, would, among other things, require the Government Operations Agency, the Department of Technology, and the Office of Data and Innovation to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, as prescribed. The bill would require, as often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities to perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
This bill would also require an automated decisionmaking system, as defined, utilized by a state agency or department to be evaluated for risk potential before adoption, as specified.
Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, defined to mean audio or visual content that has been generated or manipulated by artificial intelligence that would falsely appear to be authentic or truthful and that features depictions of people appearing to say or do things they did not say or do without their consent, on state government, California-based businesses, and residents of the state.
This bill, the Artificial Intelligence Accountability Act, would, among other things, require the Government Operations Agency, the Department of Technology, and the Office of Data and Innovation to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes certain items, including an examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state, and would require those entities to update the report, as prescribed. The bill would require, as often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities to perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
This bill would also require a state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
This bill would also require an automated decisionmaking system, as defined, utilized by a state agency or department to be evaluated for risk potential before adoption, as specified.
This act shall be known as the Generative Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Generative Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of GenAI systems. The public has the right to know when they are interacting with GenAI being used by the state and to have an accessible identification of that interaction.
(e)The Legislature recognizes that the use of GenAI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all GenAI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies.
(g)Public-private cross-sectional partnerships should be established to train students to meet the GenAI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train GenAI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal.
(h)State agencies, departments, and boards should utilize their authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems.
11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Generative artificial intelligence or GenAI means an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio that emulates the structure and characteristics of the systems training data.
(c)Person means a natural person.
(d)Report means the report to the Governor required by Executive Order No. N-12-23.
11549.65.(a)The Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the Department of Human Resources, shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state exclusive employee representatives.
(b)(1)The Office of Emergency Services shall, as appropriate, perform a risk analysis of potential threats posed by the use of GenAI to Californias critical infrastructure, including those that could lead to mass casualty events.
(2)The analysis required by paragraph (1) shall be provided to the Governor, and, if appropriate, shall include recommendations reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(3)A high-level summary of the analysis required by paragraph (1) shall be submitted annually to the Legislature.
(c)Any state agency or department shall consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys policies for public sector GenAI procurement.
(d)Legal counsel for any state agency or department shall consider any potential impact of GenAI on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.A state agency or department that utilizes GenAI to directly communicate with a person regarding government services and benefits shall ensure that those communications include both of the following:
(a)A disclaimer that indicates to the person that the communication was generated by GenAI.
(1)For written communications involving physical and digital media, including letters, email, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
(2)For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
(3)For audio communications, the disclaimer shall be provided verbally at the start and end of the interaction.
(4)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(b)Information, or a link to an internet website containing information, describing how the person may contact a human employee of the state agency or department.
This act shall be known as the Generative Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Generative Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of GenAI systems. The public has the right to know when they are interacting with GenAI being used by the state and to have an accessible identification of that interaction.
(e)The Legislature recognizes that the use of GenAI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all GenAI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies.
(g)Public-private cross-sectional partnerships should be established to train students to meet the GenAI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train GenAI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal.
(h)State agencies, departments, and boards should utilize their authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems.
11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Generative artificial intelligence or GenAI means an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio that emulates the structure and characteristics of the systems training data.
(c)Person means a natural person.
(d)Report means the report to the Governor required by Executive Order No. N-12-23.
11549.65.(a)The Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the Department of Human Resources, shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state exclusive employee representatives.
(b)(1)The Office of Emergency Services shall, as appropriate, perform a risk analysis of potential threats posed by the use of GenAI to Californias critical infrastructure, including those that could lead to mass casualty events.
(2)The analysis required by paragraph (1) shall be provided to the Governor, and, if appropriate, shall include recommendations reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(3)A high-level summary of the analysis required by paragraph (1) shall be submitted annually to the Legislature.
(c)Any state agency or department shall consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys policies for public sector GenAI procurement.
(d)Legal counsel for any state agency or department shall consider any potential impact of GenAI on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.A state agency or department that utilizes GenAI to directly communicate with a person regarding government services and benefits shall ensure that those communications include both of the following:
(a)A disclaimer that indicates to the person that the communication was generated by GenAI.
(1)For written communications involving physical and digital media, including letters, email, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
(2)For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
(3)For audio communications, the disclaimer shall be provided verbally at the start and end of the interaction.
(4)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(b)Information, or a link to an internet website containing information, describing how the person may contact a human employee of the state agency or department.
This act shall be known as the Generative Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Generative Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of GenAI systems. The public has the right to know when they are interacting with GenAI being used by the state and to have a clear and conspicuous an accessible identification of that interaction.
(e)The Legislature recognizes that the use of GenAI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all GenAI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies. Further, state leadership on adopting these standards and best practices should encourage the private sector to adopt these practices and safeguards.
(g)Public and private colleges and universities should collaborate with the private sector and relevant state agencies Public-private cross-sectional partnerships should be established to train students to meet the GenAI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train GenAI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal, including adopting appropriate job classifications and incentive programs. goal.
(h)State agencies, departments, and boards should utilize their full range of authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems. They should also clarify any responsibility of regulated entities to conduct due diligence with respect to third-party GenAI services they use and emphasize or clarify requirements and expectations related to the transparency of AI models and regulated entities ability to explain their use of AI models.
(i)The California Privacy Protection Agency should utilize its existing authority to promulgate automated decisionmaking technology regulations.11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)Generative artificial intelligence or GenAI means an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio that emulates the structure and characteristics of the systems training data.
(c)Person means a natural person.
(d)Report means the State of California Benefits and Risk of Generative Artificial Intelligence Report report to the Governor required by Section 11549.65. Executive Order No. N-12-23.
11549.65.(a)(1)The Government Operations Agency, the Department of Technology, the Office of Data and Innovation, and the California Privacy Protection Agency shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:(A)An examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state.(B)An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of GenAI to make a consequential decision affecting access to essential goods and services.(C)An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, data privacy, and the economy. (2)11549.65.(a)The Government Operations Agency, the Department of Technology, under the guidance of the Government Operations Agency, the Office of Data and Innovation, and the California Privacy Protection Agency Department of Human Resources, shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees. exclusive employee representatives.
(b)(1)(A)The Office of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center shall every two years Services shall, as appropriate, perform a joint risk analysis of potential threats posed by the use of GenAI to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. events.
(B)The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.(2)The analysis required by paragraph (1) shall be provided to the Legislature and the Governor, and, if appropriate, shall include recommendations reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(3)The A high-level summary of the analysis required by paragraph (1) shall be submitted annually to the Legislature in compliance with Section 9795. Legislature.
(c)(1)(A)The Government Operations Agency, the Department of General Services, the Department of Technology, the California Cybersecurity Integration Center, and the California Privacy Protection Agency shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services. (B)The guidelines required by this paragraph shall incorporate guidance from the White House publication titled Blueprint for an AI Bill of Rights and the National Institute of Standards and Technologys AI Risk Management Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by GenAI. (C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, the California Cybersecurity Integration Center, and the California Privacy Protection Agency shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.(d)(1)(A)The Government Operations Agency, the Department of Technology, the Office of Data and Innovation, and the California Privacy Protection Agency shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.(B)The guidelines required by this subdivision shall inform whether and how a state agency or department deploys a particular GenAI tool.(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this subdivision.(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of Technology, the Office of Data and Innovation, and the California Privacy Protection Agency shall consult with academia, industry experts, and organizations that represent state government employees.(e)The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the states project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).(f)(1)To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to the Department of Technology, which shall administer the inventory. (2)A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).(g)(c)Any state agency or department shall consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys guidelines policies for public sector GenAI procurement.
(h)(1)The Government Operations Agency shall collaborate with the Department of Technology to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including Department of Technology-approved environments to test those pilot projects. (2)An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate GenAI tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use GenAI consistent with state guidelines.(3)An environment created pursuant to this subdivision shall measure all of the following:(A)How GenAI can improve Californians experience with, and access to, government services.(B)How GenAI can support state employees in the performance of their duties.(C)Any domain-specific impacts to be measured by the state agency or department, including job displacement.(D)How to mitigate risks of harms associated with the use of GenAI.(E)Any pilot projects conducted pursuant to this subdivision shall be consistent with the requirements of Item 0511-001-0001 of Section 2.00 of the Budget Act of 2024.(i)The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, the California Cybersecurity Integration Center, and the California Privacy Protection Agency shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.(j)A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy by complying with both of the following:(1)The Government Operations Agency, the Department of Technology, and the California Privacy Protection Agency shall make available trainings for state government worker use of state-approved GenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, privacy risks, and biases of GenAI, while enforcing applicable state laws and policies. If appropriate, the Government Operations Agency, the Department of Technology, and the California Privacy Protection Agency shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.(2)The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of GenAI on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.(k)(d)Legal counsel for any state agency or department shall consider any potential impact of GenAI on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.A state agency or department that utilizes GenAI to directly communicate with a person regarding government services and benefits shall ensure that those communications include both of the following:
(a)A disclaimer that indicates to the person that the communication was generated by GenAI.
(1)For written communications involving physical and digital media, including letters, email, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
(2)For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
(3)For audio communications, the disclaimer shall be provided verbally at the start and end of the interaction.
(4)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(b)Clear instructions Information, or a link to an internet website containing clear instructions information, describing how the person may contact a human employee of the state agency or department.
This act shall be known as the Generative Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Generative Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of AI GenAI systems. The public has the right to know when they are interacting with AI GenAI being used by the state and to have a clear and conspicuous identification of that interaction.
(e)The Legislature recognizes that the use of AI GenAI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all AI GenAI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies. Further, state leadership on adopting these standards and best practices should encourage the private sector to adopt these practices and safeguards.
(g)Public and private colleges and universities should collaborate with the private sector and relevant state agencies to train students to meet the AI GenAI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train AI GenAI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal, including adopting appropriate job classifications and incentive programs.
(h)State agencies, departments, and boards should utilize their full range of authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems. They should also clarify any responsibility of regulated entities to conduct due diligence with respect to third-party AI GenAI services they use and emphasize or clarify requirements and expectations related to the transparency of AI models and regulated entities ability to explain their use of AI models.
(i)The California Privacy Protection Agency should utilize its existing authority to promulgate automated decisionmaking technology regulations.
11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that, that varies in its level of autonomy and that can, for explicit or implicit objectives, infers, infer from the input it receives, receives how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy. environments.
(b)Generative artificial intelligence or GenAI means the class of an artificial intelligence models that emulate the structure and characteristics of input data in order to can generate derived synthetic content, including text, images, videos, audio, text, and other digital content. video, and audio that emulates the structure and characteristics of the systems training data.
(c)Person means a natural person.
(d)Report means the State of California Benefits and Risk of Generative Artificial Intelligence Report required by Section 11549.65.
11549.65.(a)(1)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation Innovation, and the California Privacy Protection Agency shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
(A)An examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence GenAI tools by the state.
(B)An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of generative artificial intelligence GenAI to make a consequential decision affecting access to essential goods and services.
(C)An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, data privacy, and the economy.
(2)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation Innovation, and the California Privacy Protection Agency shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
(b)(1)(A)As often as is deemed appropriate by the Director The Office of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities Center shall perform every two years perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence GenAI to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
(B)The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
(2)The analysis required by paragraph (1) shall be provided to the Legislature and the Governor, and, if appropriate, public recommendations shall be made include recommendations reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(3)The analysis required by paragraph (1) shall be submitted to the Legislature in compliance with Section 9795.
(c)(1)(A)The Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center Center, and the California Privacy Protection Agency shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of generative artificial intelligence, GenAI, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
(B)The guidelines required by this paragraph shall build on incorporate guidance from the White House publication titled Blueprint for an AI Bill of Rights and the National Institute of Standards and Technologys AI Risk Management Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligence. GenAI.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center Center, and the California Privacy Protection Agency shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(d)(1)(A)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation Innovation, and the California Privacy Protection Agency shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a generative artificial intelligence GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
(B)The guidelines required by this paragraph subdivision shall inform whether and how a state agency or department deploys a particular generative artificial intelligence GenAI tool.(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph. subdivision.(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center Privacy Protection Agency shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so consult with academia, industry experts, and organizations that represent state government employees.(e)The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the states project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).
(f)(1)To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of generative artificial intelligence GenAI within the agency or department to the Department of Technology, which shall administer the inventory.
(2)A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).
(g)Any state agency or department shall consider procurement and enterprise use opportunities in which generative artificial intelligence GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys guidelines for public sector generative artificial intelligence GenAI procurement.
(h)(1)The Government Operations Agency shall collaborate with the Department of Technology shall to establish and maintain the infrastructure to conduct pilot projects of generative artificial intelligence GenAI projects, including Department of Technology-approved environments to test those pilot projects.
(2)An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate generative artificial intelligence GenAI tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use generative artificial intelligence GenAI consistent with state guidelines.
(3)An environment created pursuant to this subdivision shall measure both all of the following:
(A)How generative artificial intelligence GenAI can improve Californians experience with, and access to, government services.
(B)How generative artificial intelligence GenAI can support state employees in the performance of their duties in addition to any duties.
(C)Any domain-specific impacts to be measured by the state agency or department. department, including job displacement.
(D)How to mitigate risks of harms associated with the use of GenAI.
(E)Any pilot projects conducted pursuant to this subdivision shall be consistent with the requirements of Item 0511-001-0001 of Section 2.00 of the Budget Act of 2024.
(i)The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center Center, and the California Privacy Protection Agency shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.
(j)A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the generative artificial intelligence GenAI economy by complying with both of the following:
(1)The Government Operations Agency, the Department of Technology, and any other agencies deemed necessary the California Privacy Protection Agency shall make available trainings for state government worker use of state-approved generative artificial intelligence GenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, privacy risks, and biases of generative artificial intelligence, GenAI, while enforcing public privacy and applicable state laws and policies. If appropriate, the Government Operations Agency, the Department of Technology Technology, and any other agency or department deemed necessary the California Privacy Protection Agency shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.
(2)The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of generative artificial intelligence GenAI on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
(k)Legal counsel for any state agency or department shall consider any potential impact of generative artificial intelligence GenAI on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.(a)A state agency or department that utilizes generative artificial intelligence GenAI to directly communicate with a person, either through an online interface or telephonically, person shall clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence. ensure that those communications include both of the following:
(a)A disclaimer that indicates to the person that the communication was generated by GenAI.
(1)For written communications involving physical and digital media, including letters, email, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
(2)For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
(3)For audio communications, the disclaimer shall be provided verbally at the start and end of the interaction.
(4)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(b)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person shall provide on the state agencys or departments internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from Clear instructions or a link to an internet website containing clear instructions describing how the person may contact a human employee of the state agency or department.
This act shall be known as the Generative Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of AI systems. The public has the right to know when they are interacting with AI being used by the state and to have a clear and conspicuous identification of that interaction.
(e)The Legislature recognizes that the use of AI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all AI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies. Further, state leadership on adopting these standards and best practices should encourage the private sector to adopt these practices and safeguards.
(g)Public and private colleges and universities should collaborate with the private sector and relevant state agencies to train students to meet the AI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train AI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal, including adopting appropriate job classifications and incentive programs.
(h)State agencies, departments, and boards should utilize their full range of authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems. They should also clarify any responsibility of regulated entities to conduct due diligence with respect to third-party AI services they use and emphasize or clarify requirements and expectations related to the transparency of AI models and regulated entities ability to explain their use of AI models.
(i)The California Privacy Protection Agency should utilize its existing authority to promulgate automated decisionmaking technology regulations.
11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(b)Generative artificial intelligence means the class of artificial intelligence models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
(c)Person means a natural person.
(d)Report means the State of California Benefits and Risk of Generative Artificial Intelligence Report required by Section 11549.65.
11549.65.(a)(1)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
(A)An examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state.
(B)An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of generative artificial intelligence to make a consequential decision affecting access to essential goods and services.
(C)An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
(2)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
(b)(1)(A)As often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities shall perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
(B)The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
(2)The analysis required by paragraph (1) shall be provided to the Governor, and, if appropriate, public recommendations shall be made reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(c)(1)(A)The Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of generative artificial intelligence, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
(B)The guidelines required by this paragraph shall build on guidance from the White House publication titled Blueprint for an AI Bill of Rights and the National Institute of Standards and Technologys AI Risk Management Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligence.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(d)(1)(A)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a generative artificial intelligence tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
(B)The guidelines required by this paragraph shall inform whether and how a state agency or department deploys a particular generative artificial intelligence tool.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(e)The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the states project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).
(f)(1)To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of generative artificial intelligence within the agency or department to the Department of Technology, which shall administer the inventory.
(2)A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).
(g)Any state agency or department shall consider procurement and enterprise use opportunities in which generative artificial intelligence can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys guidelines for public sector generative artificial intelligence procurement.
(h)(1)The Department of Technology shall establish and maintain the infrastructure to conduct pilot projects of generative artificial intelligence projects, including Department of Technology-approved environments to test those pilot projects.
(2)An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate generative artificial intelligence tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use generative artificial intelligence consistent with state guidelines.
(3)An environment created pursuant to this subdivision shall measure both of the following:
(A)How generative artificial intelligence can improve Californians experience with, and access to, government services.
(B)How generative artificial intelligence can support state employees in the performance of their duties in addition to any domain-specific impacts to be measured by the state agency or department.
(i)The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.
(j)A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the generative artificial intelligence economy by complying with both of the following:
(1)The Government Operations Agency, the Department of Technology, and any other agencies deemed necessary shall make available trainings for state government worker use of state-approved generative artificial intelligence tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of generative artificial intelligence, while enforcing public privacy and applicable state laws and policies. If appropriate, the Department of Technology and any other agency or department deemed necessary shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.
(2)The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of generative artificial intelligence on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
(k)Legal counsel for any state agency or department shall consider any potential impact of generative artificial intelligence on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.(a)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, shall clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
(b)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person shall provide on the state agencys or departments internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
This act shall be known as the Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of AI systems. The public has the right to know when they are interacting with AI being used by the state and to have a clear and conspicuous identification of that interaction.
(e)The Legislature recognizes that the use of AI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all AI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies. Further, state leadership on adopting these standards and best practices should encourage the private sector to adopt these practices and safeguards.
(g)Public colleges and universities should collaborate with the private sector and relevant state agencies to train students to meet the AI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train AI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal, including adopting appropriate job classifications and incentive programs.
(h)State agencies, departments, and boards should utilize their full range of authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems. They should also clarify any responsibility of regulated entities to conduct due diligence with respect to third-party AI services they use and emphasize or clarify requirements and expectations related to the transparency of AI models and regulated entities ability to explain their use of AI models.
(i)The California Privacy Protection Agency should utilize its existing authority to promulgate automated decisionmaking technology regulations.
11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.(c)(b)Generative artificial intelligence means the class of artificial intelligence models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
(d)High-risk automated decision system means an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. (e)(c)Person means a natural person.
(f)(d)Report means the State of California Benefits and Risk of Generative Artificial Intelligence Report required by Section 11549.65.
11549.65.(a)(1)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
(A)An examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state.
(B)An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of generative artificial intelligence to make a consequential decision affecting access to essential goods and services.
(C)An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
(2)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
(b)(1)(A)As often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities shall perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
(B)The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
(2)The analysis required by paragraph (1) shall be provided to the Governor, and, if appropriate, public recommendations shall be made reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(c)(1)(A)The Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of generative artificial intelligence, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
(B)The guidelines required by this paragraph shall build on guidance from the White House publication titled Blueprint for an Al AI Bill of Rights and the National Institute of Standards and Technologys Al AI Risk Management Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligence.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(d)(1)(A)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a generative artificial intelligence tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
(B)The guidelines required by this paragraph shall inform whether and how a state agency or department deploys a particular generative artificial intelligence tool.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(e)The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the states project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).
(f)(1)To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of generative artificial intelligence within the agency or department to the Department of Technology, which shall administer the inventory.
(2)A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).
(g)Any state agency or department shall consider procurement and enterprise use opportunities in which generative artificial intelligence can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys guidelines for public sector generative artificial intelligence procurement.
(h)(1)The Department of Technology shall establish and maintain the infrastructure to conduct pilot projects of generative artificial intelligence projects, including Department of Technology-approved environments to test those pilot projects.
(2)An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate generative artificial intelligence tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use generative artificial intelligence consistent with state guidelines.
(3)An environment created pursuant to this subdivision shall measure both of the following:
(A)How generative artificial intelligence can improve Californians experience with, and access to, government services.
(B)How generative artificial intelligence can support state employees in the performance of their duties in addition to any domain-specific impacts to be measured by the state agency or department.
(i)The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.
(j)A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the generative artificial intelligence economy by complying with both of the following:
(1)The Government Operations Agency, the Department of Technology, and any other agencies deemed necessary shall make available trainings for state government worker use of state-approved generative artificial intelligence tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of generative artificial intelligence, while enforcing public privacy and applicable state laws and policies. If appropriate, the Department of Technology and any other agency or department deemed necessary shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.
(2)The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of generative artificial intelligence on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
(k)Legal counsel for any state agency or department shall consider any potential impact of generative artificial intelligence on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.(a)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, shall clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
(b)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person shall provide on the state agencys or departments internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
11549.67.(a)(1)An automated decisionmaking system utilized by a state agency or department shall be evaluated for risk potential before adoption. (2)A high-risk automated decision system shall receive appropriate consultation, testing, risk identification, and mitigation consistent with this chapter and shall not be adopted or utilized without prior approval of the director of a state agency or department or that persons designee before being adopted and utilized by a state agency or department.(b)A high-risk automated decision system that is utilized by a state agency or department shall receive ongoing monitoring and clear organizational oversight.This act shall be known as the Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of AI systems. The public has the right to know when they are interacting with AI being used by the state and to have a clear and conspicuous identification of that interaction.
(e)The Legislature recognizes that the use of AI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all AI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies. Further, state leadership on adopting these standards and best practices should encourage the private sector to adopt these practices and safeguards.
(g)Public colleges and universities should collaborate with the private sector and relevant state agencies to train students to meet the AI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train AI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal, including adopting appropriate job classifications and incentive programs.
(h)State agencies, departments, and boards should utilize their full range of authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems. They should also clarify any responsibility of regulated entities to conduct due diligence with respect to third-party AI services they use and emphasize or clarify requirements and expectations related to the transparency of AI models and regulated entities ability to explain their use of AI models.
(i)The California Privacy Protection Agency should utilize its existing authority to promulgate automated decisionmaking technology regulations.
11549.64.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(a)(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(b)(c)Generative artificial intelligence means the class of artificial intelligence models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
(c)(d)High-risk automated decision system means an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.
(d)(e)Person means a natural person.
(e)(f)Report means the State of California Benefits and Risk of Generative Artificial Intelligence Report required by Section 11549.65.
11549.65.(a)(1)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
(A)An examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state.
(B)An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of generative artificial intelligence to make a consequential decision affecting access to essential goods and services.
(C)An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
(2)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
(b)(1)(A)As often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities shall perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
(B)The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
(2)The analysis required by paragraph (1) shall be provided to the Governor, and, if appropriate, public recommendations shall be made reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(c)(1)(A)The Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of generative artificial intelligence, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
(B)The guidelines required by this paragraph shall build on guidance from the White House publication titled Blueprint for an Al Bill of Rights and the National Institute of Standards and Technologys Al Risk Management Framework Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligence.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(d)(1)(A)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a generative artificial intelligence tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
(B)The guidelines required by this paragraph shall inform whether and how a state agency or department deploys a particular generative artificial intelligence tool.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(e)The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the states project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).
(f)(1)To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of generative artificial intelligence within the agency or department to the Department of Technology, which shall administer the inventory.
(2)A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).
(g)Any state agency or department shall consider procurement and enterprise use opportunities in which generative artificial intelligence can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys guidelines for public sector generative artificial intelligence procurement.
(h)(1)The Department of Technology shall establish and maintain the infrastructure to conduct pilot projects of generative artificial intelligence projects, including Department of Technology-approved environments to test those pilot projects.
(2)An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate generative artificial intelligence tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use generative artificial intelligence consistent with state guidelines.
(i)(1)By July 1, ____, any state agency or department shall consider pilot projects of generative artificial intelligence applications, in consultation with organizations that represent state government employees and appropriate experts from academia and industry. (2)A pilot project described by paragraph (1) shall measure both of the following:(3)An environment created pursuant to this subdivision shall measure both of the following:
(A)How generative artificial intelligence can improve Californians experience with, and access to, government services.
(B)How generative artificial intelligence can support state employees in the performance of their duties in addition to any domain-specific impacts to be measured by the state agency or department.
(j)(i)The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.
(k)(j)A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the generative artificial intelligence economy by complying with both of the following:
(1)The Government Operations Agency, the Department of Technology, and any other agencies deemed necessary shall make available trainings for state government worker use of state-approved generative artificial intelligence tools to achieve equitable outcomes and to identity identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of generative artificial intelligence, while enforcing public privacy and applicable state laws and policies. If appropriate, the Department of Technology and any other agency or department deemed necessary shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.
(2)The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of generative artificial intelligence on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
(l)(k)Legal counsel for any state agency or department shall consider any potential impact of generative artificial intelligence on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.(a)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, shall clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
(b)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person shall provide on the state agencys or departments internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
11549.67.(a)(1)An automated decisionmaking system utilized by a state agency or department shall be evaluated for risk potential before adoption.
(2)A highrisk high-risk automated decision system shall receive appropriate consultation, testing, risk identification, and mitigation consistent with this chapter and shall not be adopted or utilized without prior approval of the director of a state agency or department or that persons designee before being adopted and utilized by a state agency or department.
(b)A highrisk high-risk automated decision system that is utilized by a state agency or department shall receive ongoing monitoring and clear organizational oversight.
This act shall be known as the Artificial Intelligence Accountability Act.
SEC. 2.Chapter 5.9 (commencing with Section 11549.63) is added to Part 1 of Division 3 of Title 2 of the Government Code, to read:5.9.Artificial Intelligence Tools11549.63.The Legislature finds and declares all of the following:
(a)The Legislature recognizes the tremendous potential of artificial intelligence (AI) to improve the lives of its citizens and the functioning of government. However, the Legislature also recognizes that the use of AI must be guided by principles of fairness, transparency, privacy, and accountability to ensure that the rights and opportunities of all Californians are protected in the age of artificial intelligence.
(b)The Legislature further recognizes that generative artificial intelligence (GenAI) enables significant, beneficial uses through its unique capabilities, but GenAI raises novel risks compared to conventional AI across critical areas, including democratic and legal processes, biases and equity, public health and safety, and the economy, and requires measures to address insufficiently guarded governmental systems and unintended or emergent harmful effects from this technology. Additionally, because humans have explicit and implicit biases built into our society, GenAI has the capacity to amplify these biases as it learns from input data. Therefore, it is imperative to consider the implications on Californians of, among other categories, different regions, income, races, ethnicities, gender, ages, religions, abilities, and sexual orientation for all GenAI inputs, outputs, and products for both prioritizing implementations that may promote equity and guarding against bias and other negative impacts.
(c)No individual or group should be discriminated against on the basis of race, gender, age, religion, sexual orientation, or any other protected characteristic in the design, development, deployment, or use of AI systems. The unprecedented speed of innovation and deployment of GenAI technologies necessitates proactive guardrails to protect against potential risks or malicious uses, including, but not limited to, bioterrorism, cyberattacks, disinformation, deception, violation of privacy, and discrimination or bias.
(d)The Legislature affirms the importance of transparency in the use of AI systems. The public has the right to know when they are interacting with AI being used by the state and to have a clear and conspicuous identification of that interaction.
(e)The Legislature recognizes that the use of AI systems must be consistent with the protection of privacy and civil liberties and must be guided by a commitment to equity and social justice. It is the intent of the Legislature in enacting this legislation that all AI systems be designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and minimizes bias and promotes equitable outcomes for all Californians.
(f)This act, in addition to the 2022 White House Blueprint for an AI Bill of Rights, executive guidance from the governor, statutory or regulatory requirements, and evolving best practices should guide the decisionmaking of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulations of automated decisionmaking technologies. Further, state leadership on adopting these standards and best practices should encourage the private sector to adopt these practices and safeguards.
(g)Public colleges and universities should collaborate with the private sector and relevant state agencies to train students to meet the AI workforce development needs of the state, including providing instruction on AI and related ethical, privacy, and security considerations while advancing research on best practices. Further, there is the need for the state to recruit, retain, and train AI professionals in certain state jobs, and agencies should collaborate to facilitate a pipeline and infrastructure to accomplish that goal, including adopting appropriate job classifications and incentive programs.
(h)State agencies, departments, and boards should utilize their full range of authority to protect consumers, patients, passengers, and students from fraud, discrimination, and threats to privacy and to address other risks that may arise from the use of GenAI, including risks to financial stability. State agencies, departments, and boards should consider rulemaking and emphasize or clarify if existing regulations and guidance apply to GenAI or other automated decisionmaking systems. They should also clarify any responsibility of regulated entities to conduct due diligence with respect to third-party AI services they use and emphasize or clarify requirements and expectations related to the transparency of AI models and regulated entities ability to explain their use of AI models.
(i)The California Privacy Protection Agency should utilize its existing authority to promulgate automated decisionmaking technology regulations.
11549.64.As used in this chapter:
(a)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(b)Generative artificial intelligence means the class of artificial intelligence models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
(c)High-risk automated decision system means an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.
(d)Person means a natural person.
(e)Report means the State of California Benefits and Risk of Generative Artificial Intelligence Report required by Section 11549.65.
11549.65.(a)(1)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
(A)An examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligence tools by the state.
(B)An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of generative artificial intelligence to make a consequential decision affecting access to essential goods and services.
(C)An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
(2)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
(b)(1)(A)As often as is deemed appropriate by the Director of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities shall perform a joint risk analysis of potential threats posed by the use of generative artificial intelligence to Californias critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
(B)The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
(2)The analysis required by paragraph (1) shall be provided to the Governor, and, if appropriate, public recommendations shall be made reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(c)(1)(A)The Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of generative artificial intelligence, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
(B)The guidelines required by this paragraph shall build on guidance from the White House publication titled Blueprint for an Al Bill of Rights and the National of Standards and Technologys Al Risk Management Framework and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligence.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(d)(1)(A)The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a generative artificial intelligence tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
(B)The guidelines required by this paragraph shall inform whether and how a state agency or department deploys a particular generative artificial intelligence tool.
(C)The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2)For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(e)The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the states project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).
(f)(1)To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of generative artificial intelligence within the agency or department to the Department of Technology, which shall administer the inventory.
(2)A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).
(g)Any state agency or department shall consider procurement and enterprise use opportunities in which generative artificial intelligence can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technologys guidelines for public sector generative artificial intelligence procurement.
(h)(1)The Department of Technology shall establish and maintain the infrastructure to conduct pilot projects of generative artificial intelligence projects, including Department of Technology-approved environments to test those pilot projects.
(2)An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate generative artificial intelligence tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use generative artificial intelligence consistent with state guidelines.
(i)(1)By July 1, ____, any state agency or department shall consider pilot projects of generative artificial intelligence applications, in consultation with organizations that represent state government employees and appropriate experts from academia and industry.
(2)A pilot project described by paragraph (1) shall measure both of the following:
(A)How generative artificial intelligence can improve Californians experience with, and access to, government services.
(B)How generative artificial intelligence can support state employees in the performance of their duties in addition to any domain-specific impacts to be measured by the state agency or department.
(j)The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.
(k)A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the generative artificial intelligence economy by complying with both of the following:
(1)The Government Operations Agency, the Department of Technology, and any other agencies deemed necessary shall make available trainings for state government worker use of state-approved generative artificial intelligence tools to achieve equitable outcomes and to identity and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of generative artificial intelligence, while enforcing public privacy and applicable state laws and policies. If appropriate, the Department of Technology and any other agency or department deemed necessary shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.
(2)The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of generative artificial intelligence on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
(l)Legal counsel for any state agency or department shall consider any potential impact of generative artificial intelligence on regulatory issues under the respective agencys or departments authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
11549.66.(a)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person, either through an online interface or telephonically, shall clearly and in a conspicuous manner identify to that person that the persons interaction with the state agency or department is being communicated through artificial intelligence.
(b)A state agency or department that utilizes generative artificial intelligence to directly communicate with a person shall provide on the state agencys or departments internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
11549.67.(a)(1)An automated decisionmaking system utilized by a state agency or department shall be evaluated for risk potential before adoption.
(2)A highrisk automated decision system shall receive appropriate consultation, testing, risk identification, and mitigation consistent with this chapter and shall not be adopted or utilized without prior approval of the director of a state agency or department or that persons designee before being adopted and utilized by a state agency or department.
(b)A highrisk automated decision system that is utilized by a state agency or department shall receive ongoing monitoring and clear organizational oversight.
|
Date
|
Status
|
|---|---|
| Sep 29, 2024 | Chaptered by Secretary of State. Chapter 928, Statutes of 2024. |
| Sep 29, 2024 | Approved by the Governor. |
| Sep 4, 2024 | Enrolled and presented to the Governor at 4 p.m. |
| Aug 28, 2024 | Assembly amendments concurred in. (Ayes 39. Noes 0. Page 5612.) Ordered to engrossing and enrolling. |
| Aug 27, 2024 | Ordered to special consent calendar. |
| Aug 26, 2024 | In Senate. Concurrence in Assembly amendments pending. |
| Aug 26, 2024 | Read third time. Passed. (Ayes 76. Noes 0. Page 6546.) Ordered to the Senate. |
| Aug 20, 2024 | Read second time. Ordered to third reading. |
| Aug 19, 2024 | Read second time and amended. Ordered to second reading. |
| Aug 15, 2024 | From committee: Do pass as amended. (Ayes 11. Noes 0.) (August 15). |
| Aug 7, 2024 | August 7 set for first hearing. Placed on suspense file. |
| Jul 3, 2024 | Read second time and amended. Re-referred to Com. on APPR. |
| Jul 3, 2024 | From committee: Do pass as amended and re-refer to Com. on APPR. with recommendation: To consent calendar. (Ayes 11. Noes 0.) (July 2). |
| Jun 4, 2024 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on P. & C.P. |
| Jun 3, 2024 | Referred to Com. on P. & C.P. |
| May 24, 2024 | In Assembly. Read first time. Held at Desk. |
| May 24, 2024 | Read third time. Passed. (Ayes 38. Noes 0. Page 4207.) Ordered to the Assembly. |
| May 22, 2024 | Ordered to special consent calendar. |
| May 20, 2024 | Read second time. Ordered to third reading. |
| May 16, 2024 | Read second time and amended. Ordered to second reading. |
| May 16, 2024 | From committee: Do pass as amended. (Ayes 7. Noes 0. Page 3962.) (May 16). |
| May 10, 2024 | Set for hearing May 16. |
| Apr 29, 2024 | April 29 hearing: Placed on APPR suspense file. |
| Apr 22, 2024 | Set for hearing April 29. |
| Apr 17, 2024 | From committee: Do pass and re-refer to Com. on APPR with recommendation: To consent calendar. (Ayes 11. Noes 0. Page 3634.) (April 16). Re-referred to Com. on APPR. |
| Apr 10, 2024 | Read second time and amended. Re-referred to Com. on JUD. |
| Apr 9, 2024 | From committee: Do pass as amended and re-refer to Com. on JUD. (Ayes 14. Noes 0. Page 3528.) (April 9). |
| Apr 9, 2024 | Set for hearing April 16 in JUD. pending receipt. |
| Mar 14, 2024 | Set for hearing April 9. |
| Feb 14, 2024 | Referred to Coms. on G.O. and JUD. |
| Jan 4, 2024 | From printer. May be acted upon on or after February 3. |
| Jan 3, 2024 | Introduced. Read first time. To Com. on RLS. for assignment. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Aug 28, 2024 | Special Consent SB896 Dodd et al. | Senate Floor |
Passed |
|
| Aug 26, 2024 | SB 896 Dodd Senate Third Reading By Low | Assembly Floor |
Passed |
|
| Aug 15, 2024 | Do pass as amended. | Assembly Standing Committee on Appropriations |
Passed |
|
| Jul 2, 2024 | Do pass as amended, and be re-referred to the Committee on [Appropriations] with recommendation: To Consent Calendar | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
|
| May 24, 2024 | Special Consent SB896 Dodd et al. | Senate Floor |
Passed |
|
| May 16, 2024 | Do pass as amended | Senate Standing Committee on Appropriations |
Passed |
|
| Apr 29, 2024 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Apr 16, 2024 | Do pass, but first be re-referred to the Committee on [Appropriations] with the recommendation: To Consent Calendar | Senate Standing Committee on Judiciary |
Passed |
|
| Apr 9, 2024 | Do pass as amended, but first amend, and re-refer to the Committee on [Judiciary] | Senate Standing Committee on Governmental Organization |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| Oakland Privacy | ** Support | |
| Secure Justice | ** Support |
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 896 |
UNFINISHED BUSINESS
Bill No: SB 896
Author: Dodd (D), et al.
Amended: 8/19/24
Vote: 21
SENATE GOVERNMENTAL ORG. COMMITTEE: 14-0, 4/9/24
AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Jones, Nguyen, Ochoa Bogh, Padilla, Portantino, Rubio, Seyarto, Smallwood-Cuevas
NO VOTE RECORDED: Ashby, Roth
SENATE JUDICIARY COMMITTEE: 11-0, 4/16/24
AYES: Umberg, Wilk, Allen, Ashby, Caballero, Durazo, Laird, Min, Niello, Stern, Wahab
SENATE APPROPRIATIONS COMMITTEE: 7-0, 5/16/24
AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab
SENATE FLOOR: 38-0, 5/24/24
AYES: Alvarado-Gil, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Newman, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
NO VOTE RECORDED: Allen, Archuleta
ASSEMBLY FLOOR: 76-0, 8/26/24 - See last page for vote
SUBJECT: Generative Artificial Intelligence Accountability Act
SOURCE: Author
DIGEST:
This bill requires, known as The Generative Artificial Intelligence (GenAI) Accountability Act, among other things, the Department of Technology (CDT), under the guidance of various other state entities, to report to the Governor as required by Executive Order (EO) N-12-23; requires the Office of Emergency Services (OES), as appropriate, to perform a risk analysis of potential threats posed by the use of GenAI to California’s critical infrastructure, as specified; and, requires a state agency or department that utilizes GenAI to directly communicate with a person regarding government services and benefits to ensure that those communications include a disclaimer, as specified, and information describing how the person may contact a human employee, as specified.
Assembly Amendments
significantly narrowed the bill by removing various requirements listed in EO N-12-23 that were originally proposed as new statute in this bill.
ANALYSIS:
Existing law:
- Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of CDT, and requires CDT to identify, assess, and prioritize high-risk, critical information technology services and systems across state government, as determined by CDT, for modernization, stabilization, or remediation.
- Requires the Secretary of the Government Operations Agency (GovOps) to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital provenance, as specified.
- Requires EO N-12-23 GovOps, CDT, the Office of Data and Innovation (ODI), and the Governor’s Office of Business and Economic Development (GO-Biz), in collaboration with other State agencies and departments and their workforce, to draft a report to the Governor examining the most significant, potentially beneficial use cases for deployment of GenAI tools by the State, as specified.
- Requires EO N-12-23 the California Cybersecurity Integration Center (Cal-CSIC) and the California State Threat Assessment Center (STAC), both established within OES, and inclusive of CDT, the California Military Department (CMD), and the California Highway Patrol (CHP), to perform a joint risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI, including those which could lead to mass casualty events and environmental emergencies, and develop, in consultation with external experts as appropriate from civil society, academia, and industry, a strategy to assess similar potential threats to other critical infrastructure, as specified.
- Requires EO N-12-23, once the above analysis is completed, the respective agencies to provide a classified briefing to the Governor, and where appropriate and without divulging classified information, make public recommendations for further administrative actions and/or collaboration with the Legislature to guard against these potential threats and vulnerabilities, as specified.
- Requires EO N-12-23 state agencies and departments subject to the Governor’s authority to consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with GovOps, the California Department of General Services (DGS), and CDT’s guidelines for public sector GenAI procurement.
- Requires EO N-12-23 legal counsel for all State agencies, departments, and boards subject to the Governor’s authority to consider and periodically evaluate for any potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates, where appropriate, as a result of this evolving technology.
This bill:
- Defines “artificial intelligence” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- Defines “generative artificial intelligence” or “GenAI” to mean an AI system that can generate derived synthetic content, including text, images, video, and audio that emulates the structure and characteristics of the system’s data.
- Requires CDT, under the guidance of GovOps, ODI, and the Department of Human Resources (CalHR) to update the GenAI report required by EO N-12-23, as needed, to respond to significant developments and, as appropriate, consult with academia, industry experts, and organizations that represent state exclusive employee representatives.
- Requires OES to, as appropriate, perform a risk analysis of potential threats posed by the use of GenAI to California’s critical infrastructure, including those that could lead to mass casualty events.
- Requires the risk analysis to be provided to the Governor, and, if appropriate, include recommendations reflecting changes to AI technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
- Requires a high-level summary of the risk analysis be submitted annually to the Legislature.
- Requires state agencies and departments to consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with GovOps, DGS, and CDT’s policies for public sector GenAI procurement.
- Requires legal counsel for any state agency or department to consider any potential impact of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person regarding government services and benefits to ensure that those communications include both of the following:
- A disclaimer that indicates to the person that the communication was generated by GenAI.
- For written communications involving physical and digital media, including letters, email, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
- For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
- For audio communications, the disclaimer shall be provided verbally at the start and end of the interaction.
- For video communications, the disclaimer shall be prominently displayed throughout the interaction.
- Information, or a link to an internet website containing information, describing how the person may contact a human employee of the state agency or department.
- A disclaimer that indicates to the person that the communication was generated by GenAI.
Background
Author Statement. According to the author’s office, “California must commit to harnessing the transformative power of GenAI to enhance the lives of Californians and improve the efficiency and efficacy of our state government operations. At a time when technological advancements are rapidly redefining the boundaries of possibility, it is important that we approach the integration of these powerful tools with an eye towards fairness, transparency, privacy, and accountability, while balancing the need to remain a world leader in innovation – both in the private and public sector. This bill is a first step towards ensuring that as we embrace the future, we do so in a way that upholds our key values.”
Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.
Executive Order N-12-23. Noting California’s role as a global hub for AI and the natural leader in this emerging field of technology, Governor Newsom issued EO N-12-23 in September of 2023. In announcing the EO, Governor Newsom identified the need to deploy GenAI ethically and responsibly throughout state government, protect and prepare for potential harms, and remain the world’s AI leader. The EO can generally be broken up into the following elements:
- Beneficial Uses of Generative AI Report: Directs state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
- Risk-Analysis Report: Directs state agencies and departments to perform a joint-risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s (NIST) AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including CDT approved environments or “sandboxes” to test such projects.
- Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
This bill codifies a number of the items listed in EO N-12-23 including: requiring CDT, under the guidance of GovOps, ODI, and CalHR, to update the benefits and risks of GenAI report, as needed, to respond to significant developments; requiring OES to, as appropriate, perform a risk analysis of potential threats posed by the use of GenAI to California’s critical infrastructure, including those that could lead to mass casualty events; requiring state agencies and departments to consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations; and, requiring legal counsel for state agencies and departments to consider any potential impact of GenAI on regulatory issues under that respective agency’s or department’s authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
Blueprint for an AI Bill of Rights. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The five identified principles include: Safe and Effective Systems; Algorithmic Discrimination Protections; Data Privacy; Notice and Explanation; and, Human Alternatives, Consideration, and Fallback.
Specifically, the Blueprint states that “Notice and Explanation” means designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible.
Additionally, “Human Alternatives, Consideration, and Fallback” means you should be able to opt out from automated systems in favor of a human alternative, where appropriate. In some cases, a human or other alternative may be required by law. You should have access to timely human consideration and remedy by a fallback and escalation process if an automated system fails, it produces and error, or you would like to appeal or contest its impacts on you. Human consideration and fallback should be accessible, equitable, effective, maintained, accompanied by appropriate operator training, and should not impose an unreasonable burden on the public.
This bill requires any state agency or department utilizing GenAI to communicate directly with a person to clearly disclose that the communication is AI-generated. For written communications, such as letters, emails, or occasional messages, the disclaimer must be prominently displayed at the start of each communication. In continuous online interactions, like chatbots, the disclaimer must be visible throughout the interaction. For audio communications, the disclaimer must be provided verbally at both the beginning and end of the interaction, while video communications must display the disclaimer throughout. Additionally, the communication must include information or a link to a website where the person can learn how to contact a human employee of the state agency or department. This ensures transparency in AI-generated interactions and provides individuals with the option to connect with a human if necessary.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Assembly Appropriations Committee, ongoing costs (General Fund) to CDT, ODI, and CalHR of an unknown but potentially significant amount to update the GenAI Intelligence Report and consult with the stakeholders named in the bill.
Costs (General Fund) to OES of an unknown but potentially significant amount to perform the infrastructure risk analysis, make recommendations to the Governor, and provide annual summaries to the Legislature.
Costs (General Fund, special funds) of an unknown amount to each state agency that must comply with the bill’s requirements for consideration and use of GenAI. The number of agencies that will be required to fulfill these obligations is unknown but California has more than 200 agencies, including state departments, boards, commissions, bureaus, universities, and other entities that may be affected by the bill. In the aggregate statewide, these cots will be significant. Possibly absorbable costs to GovOps to provide guidance to CDT.
SUPPORT: (Verified 8/26/24)
Oakland Privacy
Secure Justice
OPPOSITION: (Verified 8/26/24)
None received
ARGUMENTS IN SUPPORT: In support of the bill Oakland Privacy, “…Writes today to express general support for codifying sections of the Governor’s AI Blueprint [Executive Order] via Senate Bill 896 and appreciates the amendments to streamline and focus the bill specifically on generative AI. The first section of Senate Bill 896 asks the Government Operations Agency, the CA Department of Technology (CDT) and the Office of Data and Innovation to collaborate on a report to the Legislature outlining governmental use of generative artificial intelligence systems, including potentially beneficial uses and purpose and an examination of possible risks from the use of generative AI.”
Further, “[t]he final section of the bill focuses on conspicuous notification of the use of generative AI in direct state communications with Californians. We are in support of that requirement and believe it can and should be implemented on the passage of the bill.
Californians are entitled to know when they are interacting with a machine, and how to move an interaction to the human sphere when it is desirable or necessary to do so.”
ASSEMBLY FLOOR VOTE: 76-0 8/26/2024
Ayes: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Connolly, Megan Dahle, Davies, Dixon, Essayli, Flora, Mike Fong, Friedman, Gabriel, Gallagher, Garcia, Gipson, Grayson, Haney, Hart, Holden, Hoover, Irwin, Jackson, Jones-Sawyer, Kalra, Lackey, Lee, Low, Lowenthal, Maienschein, Mathis, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Pacheco, Papan, Jim Patterson, Joe Patterson, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Luz Rivas, Rodriguez, Blanca Rubio, Sanchez, Santiago, Schiavo, Soria, Ta, Ting, Valencia, Villapudua, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas
Noes:
No Vote Recorded: Cervantes, Chen, Ortega
Prepared by: Brian Duke / G.O. / (916) 651-1530
8/27/24 12:23:06
**** END ****
SENATE THIRD READING
STR Bill Id:SB 896
Author:(Dodd)
As Amended Ver:August 19, 2024
Majority vote
SUMMARY
Codifies some aspects of the Governor's artificial intelligence (AI) executive order, and requires that the use of generative artificial intelligence (GenAI) for state communications be disclosed.
Major Provisions
- Requires the Department of Technology, the Office of Data and Innovation, and the Department of Human Services to update the State of California Benefits and Risk of Generative Artificial Intelligence report, required by Executive Order No. N-12-23, as needed to respond to significant developments, as well as to consult with academia, industry experts, and organizations that represent state exclusive employee representatives.
- Requires the Office of Emergency Services perform a risk analysis of potential threats posed by the use of GenAI to California's critical infrastructure, including those that could lead to mass casualty events.
- Requires the analysis required by 2) be provided to the Governor, along with policy recommendations. Requires a high-level summary of the report be provided annually to the Legislature.
- Requires state agencies and departments to consider procurement and enterprise use opportunities for GenAI. Requires legal counsel for state agencies and departments to consider potential impacts of GenAI on regulatory issues and recommend updates in light of evolving technologies.
- Requires a state agency or department that utilizes generative artificial intelligence to directly communicate with a person regarding government services and benefits to ensure that those communications include a disclaimer that indicates to the person that the communication was generated by GenAI, and information describing how the person may contact a human employee of the state agency or department.
COMMENTS
Background. California's artificial intelligence (AI) industry is currently experiencing an unprecedented economic boom, driven in large part by the recent development of generative AI (GenAI) systems capable of generating text, images, and other forms of digital media. The Governor's Executive Order N-12-23 seeks to increase the use of GenAI by the state – it requires risk analyses be performed, procurement blueprints for GenAI systems be created, beneficial uses for GenAI technologies be identified, deployment frameworks be crafted, and employee trainings be organized. The Executive Order also initiates a series of GenAI pilot projects in the Department of Technology.
This bill would codify some provisions of Executive Order N-12-23, and would additionally require that the state disclose its use of GenAI when communicating with Californians about government benefits and services.
What this bill would do. This bill would require the Department of Technology, the Office of Data and Innovation, and the Department of Human Services to update the State of California Benefits and Risk of Generative Artificial Intelligence report, required by Executive Order No. N-12-23. It would additionally require the Office of Emergency Services perform a risk analysis of potential threats posed by the use of GenAI to California's critical infrastructure, to be provided in full to the Governor and in summary form to the Legislature. Finally, this bill would requires that a state agency or department that utilizes generative artificial intelligence to directly communicate with a person regarding government services and benefits to ensure that those communications include a disclaimer that indicates to the person that the communication was generated by GenAI, and information describing how the person may contact a human employee of the state agency or department.
Analysis. The development of GenAI is creating exciting opportunities to grow California's economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns. This bill would instruct various agencies and departments to explore these safety concerns, and would create a meaningful disclosure requirement for Government's use of GenAI to communicate with Californians.
According to the Author
California must commit to harnessing the transformative power of AI and GenAI to enhance the lives of Californians and improve the efficiency and efficacy of our state government operations. At a time when technological advancements are rapidly redefining the boundaries of possibility, it is important that we approach the integration of these powerful tools with an eye towards fairness, transparency, privacy, and accountability, while balancing the need to remain a world leader in innovation – both in the private and public sector. This bill is a first step towards ensuring that as we embrace the future, we do so in a way that upholds our key values.
Arguments in Support
Secure Justice writes:
In this era of disinformation and distrust in our civic institutions, we believe such efforts like SB 896 will guide agencies working to address concerns arising from the use of AI.
Arguments in Opposition
None.
FISCAL COMMENTS
According to the Assembly Appropriations Committee,
- Ongoing costs (General Fund) to CDT, ODI, and CalHR of an unknown but potentially significant amount to update the GenAI Intelligence Report and consult with the stakeholders named in the bill.
- Costs (General Fund) to CalOES of an unknown but potentially significant amount to perform the infrastructure risk analysis, make recommendations to the Governor, and provide annual summaries to the Legislature.
- Costs (General Fund, special funds) of an unknown amount to each state agency that must comply with the bill's requirements for consideration and use of GenAI. The number of agencies that will be required to fulfill these obligations is unknown but California has more than 200 agencies, including state departments, boards, commissions, bureaus, universities, and other entities that may be affected by the bill. In the aggregate statewide, these costs will likely be significant.
- Possibly absorbable costs to GovOps to provide guidance to CDT.
VOTES
SENATE FLOOR: 38-0-2
YES: Alvarado-Gil, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Newman, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk
ABS, ABST OR NV: Allen, Archuleta
ASM PRIVACY AND CONSUMER PROTECTION: 11-0-0
YES: Bauer-Kahan, Joe Patterson, Bryan, Dixon, Hoover, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson
ASM APPROPRIATIONS: 11-0-4
YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart, Pellerin, Villapudua
ABS, ABST OR NV: Sanchez, Dixon, Jim Patterson, Ta
UPDATED
VERSION: August 19, 2024
CONSULTANT: Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0004151
Date of Hearing: August 7, 2024
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
SBPCA Bill Id:SB 896
Author:(Dodd) – As Amended Ver:July 3, 2024
Policy Committee: | Privacy and Consumer Protection | Vote: | 11 - 0 |
|
| ||
|
|
Urgency: No State Mandated Local Program: No Reimbursable: No
SUMMARY:
This bill requires specified state agencies and departments to collaborate to produce reports, risk analyses, guidelines, and infrastructure to support, evaluate, and govern procurement and use of generative artificial intelligence (GenAI) technology by state agencies. The bill also requires each state agency to inventory its current GenAI use, and, if the agency uses GenAI to communicate with the public, requires specified public disclosures.
Specifically, among other provisions, this bill:
- Requires the Government Operations Agency (GovOps), the California Department of Technology (CDT), the Office of Data and Innovation (ODI), and the California Privacy Protection Agency (Privacy Agency) to produce and, as needed, update a report that includes the most significant uses for deployment of GenAI by the state and the potential risks of those uses, among other topics.
- Requires the Office of Emergency Services (CalOES) to perform and periodically update a risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure and other critical infrastructure.
- Requires GovOps, the Department of General Services (DGS), CDT, the California Cybersecurity Integration Center within CalOES, and the Privacy Agency to develop, maintain, and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI.
- Requires GovOps, CDT, ODI, and the Privacy Agency to develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact adopting a GenAI tool may have on vulnerable communities.
- Requires GovOps, the DGS, and CDT to update, as needed, the state’s project approval, procurement, and contract terms.
- Requires each state agency and department to conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to CDT, upon receiving a request from GovOps or CDT. Requires a state agency or department to appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting results of the inventory.
- Requires GovOps to collaborate with CDT to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including CDT-approved environments to test those pilot projects. Requires an environment to be available to any state agency or department to help evaluate GenAI tools and services.
- Requires GovOps, CDT, and the Privacy Agency to make available trainings for state government worker use of state-approved GenAI tools.
- Requires GovOps to consult with appropriate state agencies and organizations that represent state government employees, to establish criteria to evaluate the impact of GenAI on the state government workforce.
- Requires legal counsel for each state agency or department to consider any potential impacts of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates to GovOps, as appropriate.
- Requires each state agency or department that uses GenAI to directly communicate with a person to ensure its communications include a disclaimer that indicates the communication was generated by GenAI, and clear instructions describing how the person may contact a human employee of the state agency or department.
FISCAL EFFECT:
- Costs (General Fund) to CDT for additional staff and consulting. In the first year of implementation, CDT anticipates costs of $508,000 for two temporary staff positions, $1.5 million for GenAI external consultants, and $700,000 for GenAI talent practices to train state employees. In subsequent years, CDT anticipates annual ongoing costs of $3.4 million for 14 additional staff positions and $500,000 for GenAI talent practices.
CDT notes it is challenging to hire GenAI talent in the public sector since salaries are often two or three times higher in the private sector, and private sector employers can offer bonuses, equity, and stock options. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees. These challenges and additional cost pressures will likely be faced by each state agency that experiences an increased workload due to this bill.
- Costs (General Fund) to ODI for additional staff. ODI reports costs of approximately $1.74 million annually ongoing for 10 permanent positions to conduct the benefit and risk reporting, create product management training programs, develop and manage of the vulnerable communities guidelines, and onboard new proofs of concept. ODI reports it temporarily reassigned existing staff to begin these efforts following the Governor’s Executive Order, but anticipates replacing those employees with additional permanent employees if this bill is enacted.
- GovOps reports no costs. GovOps anticipates that if this bill is enacted, CDT and ODI will receive funding for additional staffing for implementation, and GovOps will primarily provide direction on strategy and policy. However, if GovOps must ultimately hire additional staff to fulfill its obligations under the bill, associated costs would likely be at least in the hundreds of thousands of dollars annually (General Fund).
- Costs (General Fund) to the Privacy Agency for additional staff and consulting. The Privacy Agency reports costs of $710,000 annually ongoing. This estimate includes two additional full time positions (an attorney and an information officer) to develop the required reports and guidelines, and coordinate collection of training materials in collaboration with other agencies, plus $250,000 annually to contract with a technologist to provide AI expertise in the development of the reports, guidelines, and trainings. The Privacy Agency also reports it may need additional administrative resources, especially if multiple bills from this session are enacted that increase staffing at the agency.
- Costs (General Fund) to CalOES of an unknown but potentially significant amount for additional staffing to perform and update risk assessments and help develop and update procurement and contracting guidelines, among other duties.
- Costs (General Fund) to DGS of an unknown but potentially significant amount for additional staffing to help develop and update procurement and contracting guidelines, among other duties.
- Costs (General Fund, special funds) to each state agency that must comply with the bill’s requirements for evaluation and use of GenAI. The number of agencies that will be required to fulfill these obligations is unknown but California has more than 200 agencies, including state departments, boards, commissions, bureaus, universities, and other entities that may be affected by the bill. For example, the Air Resources Board reports costs of approximately $2.5 million in fiscal year (FY) 2024-25 and $2.4 million in FY 2025-26 and ongoing to comply with this bill (Air Pollution Control Fund). While not all impacted agencies will require the same level of resources, aggregate costs statewide may be in the tens of millions of dollars annually or higher.
According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years.
COMMENTS:
- Purpose. According to the author of the bill:
California must commit to harnessing the transformative power of AI and GenAI to enhance the lives of Californians and improve the efficiency and efficacy of our state government operations. At a time when technological advancements are rapidly redefining the boundaries of possibility, it is important that we approach the integration of these powerful tools with an eye towards fairness, transparency, privacy, and accountability, while balancing the need to remain a world leader in innovation – both in the private and public sector.
- Background. In 2023, Governor Newsom issued Executive Order (EO) N-12-23, which directed specified state entities “to study the development, use, and risks of AI technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government.”
This bill largely codifies the Governor’s EO, but expands on the EO in two significant ways. First, unlike the EO, the bill requires the Privacy Agency to help (a) create and update the report that includes the most significant uses for deployment of GenAI by the state and the potential risks of those uses, (b) develop and revise guidelines for public sector procurement and use of GenAI, (c) create guidelines for state entities to analyze the impact adopting a GenAI tool may have on vulnerable communities, and (d) make available trainings for state government worker use of state-approved GenAI tools. The Governor’s EO does not include a direct role for the Privacy Agency in these tasks or the other advisory duties this bill gives to the Privacy Agency. Second, unlike the EO, the bill requires each state agency that deploys GenAI to communicate directly with a member of the public to disclose its use of GenAI and provide instructions for contacting a human representative instead.
Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081
Date of Hearing: July 2, 2024
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
SBPCA Bill Id:SB 896
Author:(Dodd) – As Amended Ver:June 4, 2024
AS PROPOSED TO BE AMENDED
SENATE VOTE: 38-0
SUBJECT: Generative Artificial Intelligence Accountability Act
SYNOPSIS
California’s artificial intelligence (AI) industry is currently experiencing an unprecedented economic boom, driven in large part by the recent development of generative AI (GenAI) systems capable of generating text, images, and other forms of digital media. The Governor’s Executive Order N-12-23 seeks to increase the use of GenAI by the state – it requires risk analyses be performed, procurement blueprints for GenAI systems be created, beneficial uses for GenAI technologies be identified, deployment frameworks be crafted, and employee trainings be organized. The Executive Order also initiates a series of GenAI pilot projects in the Department of Technology.
This bill would codify many of the provisions of Executive Order N-12-23, and would additionally require that the state disclose its use of GenAI when communicating. Committee amendments strengthen the bill by involving the California Privacy Protection Agency more directly in many of its provisions, aligning pilot program language with a related Budget item, and clarifying GenAI communication disclosure requirements.
This bill is author-sponsored and supported by Oakland Privacy and Secure Justice. It has no opposition. If passed out of this Committee, this bill will next be heard by the Assembly Appropriations Committee.
SUMMARY: Codifies the Governor’s Executive Order N-12-23, related to the procurement and use of GenAI by the state, and requires that the use of GenAI for state communications be disclosed. Specifically, this bill:
- Defines “artificial intelligence” to mean “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- Defines “generative artificial intelligence” to mean “artificial intelligence that can generate derived synthetic content, including text, images, video, and audio, that emulates the structure and characteristics of the artificial intelligence’s training data.
- Requires the Government Operations Agency, the Department of Technology, the Office of Data and Innovation, and the California Privacy Protection Agency to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
- An examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state.
- An explanation of the potential risks of those uses to individuals, communities, and government workers with a focus on high-risk uses, including the use of GenAI to make a consequential decision affecting access to essential goods and services.
- An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, data privacy, and the economy.
- Requires the Government Operations Agency, the Department of Technology, the Office of Data and Innovation, and the California Privacy Protection Agency to update the report as needed to respond to significant developments, as well as to consult with academia, industry experts, and organizations that represent state government employees.
- Requires the Office of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, to perform and periodically update a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. Also requires those entities to develop a strategy to assess similar potential threats to other critical infrastructure in consultation with appropriate external experts from academia and industry. Requires this analysis to be provided to the Legislature and the Governor. Requires the named entities to consult with academia, industry experts, and organizations that represent state government employees.
- Requires the Government Operations Agency, the Department of General Services, the Department of Technology, the California Cybersecurity Integration Center, and the California Privacy Protection Agency to develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI. Requires these guidelines to incorporate guidance from a variety of sources, as specified. Requires the named entities to consult with academia, industry experts, and organizations that represent state government employees.
- Requires the Government Operations Agency, the Department of Technology, the Office of Data and Innovation, and the California Privacy Protection Agency to develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses. Requires the named entities to consult with academia, industry experts, and organizations that represent state government employees.
- These guidelines are required to inform whether and how a state agency or department deploys a particular GenAI tool.
- Requires the Government Operations Agency, the Department of General Services, and the Department of Technology to update, as needed, the state’s project approval, procurement, and contract terms to incorporate analysis and feedback.
- Requires all state agencies and departments to conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to the Department of Technology, upon receiving a request from the Government Operations Agency or the Department of Technology.
- Requires a state agency or department to appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory.
- Requires any state agency or department to consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technology’s guidelines.
- Requires the Government Operations Agency to collaborate with the Department of Technology to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including Department of Technology-approved environments to test those pilot projects.
- Requires an environment to be available to any state agency or department to help evaluate GenAI tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use GenAI consistent with state guidelines.
- Requires an environment to measure all of the following:
- How GenAI can improve Californians’ experience with, and access to, government services.
- How GenAI can support state employees in the performance of their duties.
- Any domain-specific impacts to be measured by the state agency or department, including job displacement.
- How to mitigate risks of harms associated with the use of GenAI.
- Requires any pilot projects conducted pursuant to this subdivision shall be consistent with the requirements of Item 0511-001-0001 of Section 2.00 of the Budget Act of 2024.
- Requires the Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, the California Cybersecurity Integration Center, and the California Privacy Protection Agency to engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training required by this bill.
- Requires the Government Operations Agency, the Department of Technology, and the California Privacy Protection Agency to make available trainings for state government worker use of state-approved GenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, privacy risks, and biases of GenAI, while enforcing public privacy and applicable state laws and policies. Requires the named entities to consult with academia, industry experts, and organizations that represent state government employees.
- Requires the Government Operations Agency to consult with appropriate state agencies and organizations that represent state government employees, to establish criteria to evaluate the impact of GenAI on the state government workforce, as well as to provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
- Requires legal counsel for any state agency or department to consider any potential impacts of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates to the Government Operations Agency, as appropriate.
- Requires a state agency or department that utilizes generative artificial intelligence to directly communicate with a person to ensure that those communications include both of the following:
- A disclaimer that indicates to the person that the communication was generated by GenAI.
- For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
- For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
- For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
- For video communications, the disclaimer shall be prominently displayed throughout the interaction.
- Clear instructions, or a link to an internet website containing clear instructions, describing how the person may contact a human employee of the state agency or department.
EXISTING LAW:
- Provides, pursuant to the California Constitution, that all people are by nature free and independent and have inalienable rights. Among these the fundamental right to privacy. (Cal. Const. art. I, § 1.)
- Establishes the Information Practices Act (IPA) of 1977, which generally enumerates the requirements applicable to state agencies that collect, maintain, and disclose personal information from California residents, including limitations on permissible disclosure, the rights of residents to know and access the information, and required accounting of disclosures of the information. (Civ. Code § 1798, et seq.)
- States, in the IPA, that the “right to privacy is a personal and fundamental right protected by Section 1 of Article I of the Constitution of California and by the United States Constitution and that all individuals have a right of privacy in information pertaining to them.” Further states these findings of the Legislature:
- The right to privacy is being threatened by the indiscriminate collection, maintenance, and dissemination of personal information and the lack of effective laws and legal remedies.
- The increasing use of computers and other sophisticated information technology has greatly magnified the potential risk to individual privacy that can occur from the maintenance of personal information.
- In order to protect the privacy of individuals, it is necessary that the maintenance and dissemination of personal information be subject to strict limits. (Civ. Code § 1798.1.)
- Requires that each state agency maintain in its records only personal information that is relevant and necessary to accomplish the purpose of the agency. (Civ. Code § 1798.14.)
- Requires that each agency collect personal information to the greatest extent practicable directly from the individual who is the subject of the information rather than from another source. (Civ. Code § 1798.15.)
- Prohibits an individual’s name and address from being distributed for commercial purposes, sold, or rented by an agency unless such action is specifically authorized by law. (Civ. Code § 1798.60.)
- Defines “personal information,” for purposes of the IPA, as any information that is maintained by an agency that identifies or describes an individual, including, but not limited to, the individual’s name, social security number, physical description, home address, home telephone number, education, financial matters, and medical or employment history. (Civ. Code § 1798.3(a).)
- Defines “agency”, for the purposes of the IPA, to mean every state office, officer, department, division, bureau, board, commission, or other state agency, except for the California Legislature, agencies within the judicial branch, the State Compensation Insurance Fund, and local agencies, defined to include: counties; cities, whether general law or chartered; cities and counties; school districts; municipal corporations; districts; political subdivisions; or any board, commission, or agency thereof; other local public agencies, or entities that are legislative bodies of a local agency as specified. (Civ. Code § 1798.3(b); Gov. Code § 6252(a).)
- Requires each agency to keep an accurate accounting of the date, nature, and purpose of each disclosure of a record made pursuant to specified circumstances; and requires each agency to retain that accounting for at least three years after the disclosure, or until the record is destroyed, whichever is shorter. (Civ. Code §§ 1798.25 & 1798.27.)
- Except as specified, endows each individual with the following rights: to inquire and be notified as to whether the agency maintains a record about them; to inspect all personal information in any record maintained by reference to an identifying particular of the individual; and to submit a request in writing to amend a record containing personal information pertaining to them maintained by an agency. (Civ. Code § 1798.30, et seq.)
- Requires each state agency, when it provides by contract for the operation or maintenance of records containing personal information to accomplish an agency function, to cause, consistent with its authority, the requirements of the IPA to be applied to those records; and specifies that for purposes of enforcing penalties for violations of the IPA, any contractor and any employee of the contractor, shall be considered to be an employee of an agency. (Civ. Code § 1798.19.)
- Establishes the Department of Technology within the Government Operations Agency, under the supervision of the Director of Technology. (Gov. Code § 11545(a).)
- Requires, upon appropriation by the Legislature, the Secretary of the Government Operations Agency to evaluate the following:
- The impact of the proliferation of deepfakes on state government, California-based businesses, and residents of the state.
- The risks, including privacy risks, associated with the deployment of digital content forgery technologies and deepfakes on state and local government, California-based businesses, and residents of the state.
- Potential privacy impacts of technologies allowing public verification of digital content provenance.
- The impact of digital content forgery technologies and deepfakes on civic engagement, including voters.
- The legal implications associated with the use of digital content forgery technologies, deepfakes, and technologies allowing public verification of digital content provenance.
- The best practices for preventing digital content forgery and deepfake technology to benefit the state, California-based businesses, and California residents, including exploring whether and how the adoption of a digital content provenance standard could assist with reducing the proliferation of digital content forgeries and deepfakes. (Gov. Code § 11547.5(b).)
- Requires the Secretary of the Government Operations Agency to develop a coordinated plan to accomplish all of the following:
- Investigate the feasibility of, and obstacles to, developing standards and technologies for state departments for determining digital content provenance.
- Increase the ability of internet companies, journalists, watchdog organizations, other relevant entities, and members of the public to meaningfully scrutinize and identify digital content forgeries and relay trust and information about digital content provenance to content consumers.
- Develop or identify mechanisms for content creators to cryptographically certify authenticity of original media and nondeceptive manipulations.
- Develop or identify mechanisms for content creators to enable the public to validate the authenticity of original media and nondeceptive manipulations to establish digital content provenance without materially compromising personal privacy or civil liberties. (Gov. Code § 11547.5(c).)
- Expresses the intent of the Legislature that policies and procedures developed by the Department of Technology and Department of General Services pertaining to the acquisition of information technology (IT) goods and services provide for all of the following: the expeditious and value-effective acquisition of IT goods and services to satisfy state requirements; the acquisition of IT goods and services within a competitive framework; the delegation of authority by the Department of General Services to each state agency that has demonstrated to the Department of General Services’ satisfaction the ability to conduct value-effective IT goods and services acquisitions; and the review and resolution of protests submitted by any bidders with respect to any IT goods and services acquisitions. (Pub. Con. Code § 12101.)
- Requires the Department of Technology, on or before September 1, 2024, to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decisionmaking tools that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. (Gov. Code § 11546.45.5(b).)
- Requires the Department of Technology, on or before January 1, 2025, and annually thereafter, to submit a report, as specified, of the comprehensive inventory to the Assembly Committee on Privacy and Consumer Protection and the Senate Committee on Governmental Organization. This requirement expires on January 1, 2029. (Gov. Code § 11546.45.5(d).)
FISCAL EFFECT: As currently in print, this bill is keyed fiscal.
COMMENTS:
- AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.
In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.
- Haphazard training data. There is a common saying in computer science: “garbage in, garbage out.” The performance of an AI product is directly impacted by the quality, quantity, and relevance of the data used to train it. Before training, datasets are often categorized to make them easier for AI to work with. Rigorously categorizing the data in a dataset becomes more difficult as the dataset becomes larger, but failing to organize its contents can lead to meaningless, false, or harmful outputs.
The biggest names in AI – OpenAI, Meta, and Google – understand AI’s critical need for data better than anyone else. According to a recent New York Times examination, the race to lead in the AI space has become a desperate hunt for digital data. To obtain that data, these tech companies have cut corners, ignored corporate policies and debated bending the law:
At Meta, which owns Facebook and Instagram, managers, lawyers and engineers last year discussed buying the publishing house Simon & Schuster to procure long works, according to recordings of internal meetings obtained by The Times. They also conferred on gathering copyrighted data from across the internet, even if that meant facing lawsuits. Negotiating licenses with publishers, artists, musicians and the news industry would take too long, they said.
Like OpenAI, Google transcribed YouTube videos to harvest text for its A.I. models, five people with knowledge of the company’s practices said. That potentially violated the copyrights to the videos, which belong to their creators.
Last year, Google also broadened its terms of service. One motivation for the change, according to members of the company’s privacy team and an internal message viewed by The Times, was to allow Google to be able to tap publicly available Google Docs, restaurant reviews on Google Maps and other online material for more of its A.I. products.[1]
Meta and Google are privy to some of the most sensitive information in the world. In many developing countries, Facebook effectively is the internet. A tremendous number of Californians use Google, or Google Chrome, or Google Drive, or Google Cloud, or Gmail.
In their race to obtain vast quantities of training data, major AI developers have not hesitated to move fast and break things. The Stanford Internet Observatory recently discovered that a common image training dataset known as LAION-5B contains many instances of child sexual abuse materials. Their study identified 3226 dataset entries of suspected child pornography, much of which was later confirmed as such by third parties. This dataset was built by automatically scraping the internet, and images containing child pornography were found to have originated from large, well-known websites such as Reddit, Twitter, Blogspot, and Wordpress, as well as mainstream adult sites such as XHamster and XVideos.[2]
- An AI never forgets. Just as humans cannot intentionally forget information they have learned, it is not currently possible to remove data from a trained AI. Unlike an Excel spreadsheet, which stores data in neat columns, AI stores data in the connections between “neurons” in a “neural network.” Every one of these connections is influenced by every piece of training data, and a large model like ChatGPT-4 is reported to have more than 1.7 trillion connections. It is not possible to specifically alter these connections in order to remove data without fundamentally changing the model; as a result, for data to be removed, the model must be retrained from scratch. ChatGPT-4 is estimated to have taken 4-7 months to train in the first place.
What happens when an AI is trained on extremely sensitive information – for example, an individual’s DNA sequence, or their social security number, or their intimate photos, or their immigration status? The same thing that happens when an AI is trained on any other type of information: the AI digests it, and then retains it forever. AI are fundamentally different from other forms of data storage. They are black holes in the information ecosystem, with “training” as their event horizons. Once data has crossed this threshold it cannot be removed.
- Risk management. The Senate Judiciary Committee’s analysis of SB 1047 summarizes various risk management frameworks related to the use of GenAI:
In January of 2017, AI researchers, economists, legal scholars, ethicists, and philosophers met in Asilomar, California to discuss principles for managing the responsible development of AI. The collaboration resulted in the Asilomar Principles. Aspirational rather than prescriptive, these 23 principles were intended to initiate and frame a dialogue by providing direction and guidance for policymakers, researchers, and developers. Its endorsers include 1,200 leading experts in the field of AI, including DeepMind founder Demis Hassabis and the late Stephen Hawking.[3]
The Legislature subsequently adopted ACR 215 (Kiley, Ch. 206, Stats. 2018), which added the State of California to that list by endorsing the Asilomar Principles as guiding values for the development of artificial intelligence and related public policy. In broad strokes, those principles aim to do the following:
- Research issues: create beneficial AI; direct funding toward beneficial innovation; maintain constructive and healthy exchanges between AI researchers and policymakers; promote a culture of trust, cooperation, and transparency among researchers and developers of AI; and avoid corner-cutting on safety standards.
- Ethics and values: promote safety, failure transparency, judicial transparency, and responsible innovation; align human values with innovation; protect privacy and liberty; ensure that the benefits and prosperity created by AI are broadly shared; maintain human control over AI; develop AI that supports rather than subverts social and civil processes; and avoid an AI arms race.
- Longer-term issues: avoid assumptions regarding the capabilities of AI; give AI its due attention; and recognize that its risks are potentially catastrophic or existential.
As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk
Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.
While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.[4]
More recently the Biden Administration has published its Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the design, use, and deployment of AI to protect the rights of the American public:
- Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
- Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
- Data Privacy: You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used. You should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used. Systems should not employ user experience and design decisions that obfuscate user choice or burden users with defaults that are privacy invasive. Consent should only be used to justify collection of data in cases where it can be appropriately and meaningfully given. Any consent requests should be brief, be understandable in plain language, and give you agency over data collection and the specific context of use; current hard-to-understand notice-and-choice practices for broad uses of data should be changed. Enhanced protections and restrictions for data and inferences related to sensitive domains, including health, work, education, criminal justice, and finance, and for data pertaining to youth should put you first. In sensitive domains, your data and related inferences should only be used for necessary functions, and you should be protected by ethical review and use prohibitions. You and your communities should be free from unchecked surveillance; surveillance technologies should be subject to heightened oversight that includes at least pre-deployment assessment of their potential harms and scope limits to protect privacy and civil liberties. Continuous surveillance and monitoring should not be used in education, work, housing, or in other contexts where the use of such surveillance technologies is likely to limit rights, opportunities, or access. Whenever possible, you should have access to reporting that confirms your data decisions have been respected and provides an assessment of the potential impact of surveillance technologies on your rights, opportunities, or access.
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.[5]
TechEquity, an organization committed to ensuring technology’s evolution benefits everyone equitably, has also laid out their straightforward AI Policy Principles:
- People who are impacted by AI must have agency to shape the technology that dictates their access to critical needs like employment, housing, and healthcare.
- The burden of proof must lie with developers, vendors, and deployers to demonstrate that their tools do not create harm—and regulators, as well as private [individuals], should be empowered to hold them accountable.
- Concentrated power and information asymmetries must be addressed in order to effectively regulate the technology.[6]
- Governor Gavin Newsom’s GenAI Executive Order. In September of 2023, Governor Gavin Newsom signed Executive Order N-12-23 with the stated goals of “deploy[ing] GenAI ethically and responsibly throughout state government, protect[ing] and prepar[ing] for potential harms, and remain[ing] the world’s AI leader.”[7] The executive order lists a number of provisions:
- Risk-Analysis Report: Direct state agencies and departments to perform a joint risk-analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Beneficial Uses of GenAI Report: Direct state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
- Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including California Department of Technology approved environments or “sandboxes” to test such projects.
- State Employee Training: To support California’s state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy, agencies will provide trainings for state government workers to use state-approved GenAI to achieve equitable outcomes, and will establish criteria to evaluate the impact of GenAI to the state government workforce.
- GenAI Partnership and Symposium: Establish a formal partnership with the University of California, Berkeley and Stanford University to consider and evaluate the impacts of GenAI on California and what efforts the state should undertake to advance its leadership in this industry. The state and the institutions will develop and host a joint summit in 2024 to engage in meaningful discussions about the impacts of GenAI on California and its workforce.
- Legislative Engagement: Engage with Legislative partners and key stakeholders in a formal process to develop policy recommendations for responsible use of AI, including any guidelines, criteria, reports, and/or training.
- Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
- Item 0511-001-0001 of Section 2.00 of the Budget Act of 2024. This budget item contains several provisions related to the use of GenAI by the state:
- Requires the Government Operations Agency to establish monthly GenAI meetings with designated legislative staff of the Assembly and Senate to discuss the agency’s ongoing efforts regarding the implementation of Executive Order N-12-23.
- Requires that personal information that is collected, maintained, or used under the GenAI pilot projects be treated as the term is defined in Section 1798.3 of the Civil Code (see EXISTING LAW section.)
- Requires that the Government Operations Agency only use publicly available data for GenAI pilot projects, and that if such data includes personal information, the Agency shall only use the information of persons aged 18 and older.
- Requires that the Government Operations Agency require departments participating in GenAI pilot projects to comply with procurement-related trainings.
- Requires that if the Government Operations agency collects personal information while implementing GenAI pilot projects, the agency may allow departments to use de-identified data to conduct research when justifying requests for resources should the department seek approval to expand the scope of the pilot program.
- Requires that the above provisions are only applicable for the 2024-2025 fiscal year.
- What this bill would do. This bill would effectively codify the Governor’s Executive Order N-12-23, related to the procurement and use of GenAI by the state. It would additionally require that the use of GenAI for communications be disclosed to the recipient of those communications.
- Author’s statement:
California must commit to harnessing the transformative power of AI and GenAI to enhance the lives of Californians and improve the efficiency and efficacy of our state government operations. At a time when technological advancements are rapidly redefining the boundaries of possibility, it is important that we approach the integration of these powerful tools with an eye towards fairness, transparency, privacy, and accountability, while balancing the need to remain a world leader in innovation – both in the private and public sector. This bill is a first step towards ensuring that as we embrace the future, we do so in a way that upholds our key values.
- Committee amendments. A number of proposed Committee amendments would clarify and strengthen the provisions of this bill. The first requires the California Privacy Protection Agency to be involved in several of the bill’s goals, in order to ensure that Californians’ privacy is being considered as the state develops trainings and adopts GenAI tools:
- The creation of the California Benefits and Risk of Generative Artificial Intelligence Report.
- The development of guidelines for procurement, use, and trainings for the use of GenAI.
- The development of guidelines for state agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities.
- Engagement with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees.
The second Committee amendment would align the pilot projects required by this bill with related language in Item 0511-001-0001 of Section 2.00 of the Budget Act of 2024, and would additionally require environments created to test pilot projects to additionally measure how to mitigate risks of harms associated with the use of GenAI.
The third Committee amendment would clarify how disclosing the use of GenAI to communicate should be performed for various communication modalities, including written communications, audio communications, and video communications. The introduced language aligns closely with the language of AB 3030 (Calderon, 2024).
- Full text of bill as proposed to be amended:
11549.64. As used in this chapter:
(a) “Artificial intelligence” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
(b) “Generative artificial intelligence” or “GenAI” means artificial intelligence that can generate derived synthetic content, including text, images, video, and audio, that emulates the structure and characteristics of the artificial intelligence’s training data.“Generative artificial intelligence” means= the class of artificial intelligence models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
(c) “Person” means a natural person.
(d) “Report” means the State of California Benefits and Risk of Generative Artificial Intelligence Report required by Section 11549.65.
11549.65. (a) (1) The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation, and the California Privacy Protection Agency shall produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
(A) An examination of the most significant, potentially beneficial uses for deployment of generative artificial intelligenceGenAI tools by the state.
(B) An explanation of the potential risks of the uses described in subparagraph (A) to individuals, communities, and government workers with a focus on high-risk uses, including the use of generative artificial intelligenceGenAI to make a consequential decision affecting access to essential goods and services.
(C) An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, data privacy, and the economy.
(2) The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation, and the California Privacy Protection Agency shall update the report, as needed, to respond to significant developments and shall, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
(b) (1) (A) As often as is deemed appropriate by the Director ofThe Office of Emergency Services, the California Cybersecurity Integration Center, and the State Threat Assessment Center, those entities shall perform, and periodically update, a joint risk analysis of potential threats posed by the use of generative artificial intelligenceGenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
(B) The entities described in subparagraph (A) shall develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
(2) The analysis required by paragraph (1) shall be provided to the Legislature and the Governor, and, if appropriate, public recommendations shall be madeinclude recommendations reflecting changes to artificial intelligence technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
(c) (1) (A) The Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center, and the California Privacy Protection Agency shall develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of generative artificial intelligenceGenAI, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
(B) The guidelines required by this paragraph shall build onincorporate guidance from the White House publication titled Blueprint for an AI Bill of Rights and the National Institute of Standards and Technology’s AI Risk Management Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligenceGenAI.
(C) The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2) For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center, and the California Privacy Protection Agency shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do soconsult with academia, industry experts, and organizations that represent state government employees.
(d) (1) (A) The Government Operations Agency, the Department of Technology, and the Office of Data and Innovation, and the California Privacy Protection Agency shall develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a generative artificial intelligenceGenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
(B) The guidelines required by this paragraph shall inform whether and how a state agency or department deploys a particular generative artificial intelligenceGenAI tool.
(C) The Government Operations Agency shall engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines required by this paragraph.
(2) For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
(2) For purposes of the periodic evaluation and revision required by paragraph (1), the Government Operations Agency, the Department of General Services, the Department of Technology, and the California Cybersecurity Integration Center shall consult with academia, industry experts, and organizations that represent state government employees.
(e) The Government Operations Agency, the Department of General Services, and the Department of Technology shall update, as needed, the state’s project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to subdivisions (c) and (d).
(f) (1) To assist the Government Operations Agency and the Department of Technology in their efforts to perform any periodic review and update under this section, all state agencies and departments shall, as requested by the Government Operations Agency or the Department of Technology, conduct and submit an inventory of all current high-risk uses of generative artificial intelligenceGenAI within the agency or department to the Department of Technology, which shall administer the inventory.
(2) A state agency or department shall appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described by paragraph (1) to the Department of Technology within 60 days of issuance of a request for an inventory pursuant to paragraph (1).
(g) Any state agency or department shall consider procurement and enterprise use opportunities in which generative artificial intelligenceGenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the Government Operations Agency, the Department of General Services, and the Department of Technology’s guidelines for public sector generative artificial intelligenceGenAI procurement.
(h) (1) The Government Operations Agency shall collaborate with the Department of Technology shall to establish and maintain the infrastructure to conduct pilot projects of generative artificial intelligenceGenAI projects, including Department of Technology-approved environments to test those pilot projects.
(2) An environment created pursuant to this subdivision shall be available to any state agency or department to help evaluate generative artificial intelligenceGenAI tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use generative artificial intelligenceGenAI consistent with state guidelines.
(3) An environment created pursuant to this subdivision shall measure both all of the following:
(A) How generative artificial intelligenceGenAI can improve Californians’ experience with, and access to, government services.
(B) How generative artificial intelligenceGenAI can support state employees in the performance of their duties. in addition to
(C) any Any domain-specific impacts to be measured by the state agency or department, including job displacement.
(D) How to mitigate risks of harms associated with the use of GenAI.
(4) Any pilot projects conducted pursuant to this subdivision shall be consistent with the requirements of Item 0511-001-0001 of Section 2.00 of the Budget Act of 2024.
(i) The Government Operations Agency, the Department of General Services, the Department of Technology, the Office of Data and Innovation, and the California Cybersecurity Integration Center, and the California Privacy Protection Agency shall engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this section.
(j) A state agency or department shall support the state government workforce and prepare for the next generation of skills needed to thrive in the generative artificial intelligenceGenAI economy by complying with both of the following:
(1) The Government Operations Agency, the Department of Technology, and the California Privacy Protection Agency, and any other agencies deemed necessary shall make available trainings for state government worker use of state-approved generative artificial intelligenceGenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, privacy risks, and biases of generative artificial intelligenceGenAI, while enforcing public privacy and applicable state laws and policies. If appropriate, the Government Operations Agency, the Department of Technology, and the California Privacy Protection Agency and any other agency or department deemed necessary shall collaborate with organizations that represent state government employees and industry experts on developing and providing training.
(2) The Government Operations Agency, in consultation with appropriate state agencies and organizations that represent state government employees, shall establish criteria to evaluate the impact of generative artificial intelligenceGenAI on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
(k) Legal counsel for any state agency or department shall consider any potential impact of generative artificial intelligenceGenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates to the Government Operations Agency , if as appropriate, as a result of this evolving technology..
11549.66. (a) A state agency or department that utilizes generative artificial intelligenceGenAI to directly communicate with a person, either through an online interface or telephonically, shall ensure that those communications include both of the following:
- A disclaimer that indicates to the person that the communication was generated by GenAI.
clearly and in a conspicuous manner identify to that person that the person’s interaction with the state agency or department is being communicated through artificial intelligence.(A) For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
(B) For written communications involving continuous online interactions, including interactions with chatbots, the disclaimer shall be prominently displayed throughout the interaction.
(C) For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D) For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(b2) Clear instructions, or a link to an internet website containing clear instructions, describing how the person may contact a human employee of the A state agency or department that utilizes generative artificial intelligence to directly communicate with a person shall provide on the state agency’s or department’s internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department..
- Related legislation. SB 892 (Padilla, 2024) would require the Department of Technology to create standards for the procurement of automated decision systems by the state. This bill is currently pending in this Committee.
SB 893 (Padilla, 2024) would create the California Artificial Intelligence Research Hub in the Government Operations Agency to increase lawful access to government data. This bill is currently pending in this Committee.
SB 1047 (Wiener, 2024) would create the Division of Frontier Models in the Government Operations Agency to oversee the development of the largest, most advanced artificial intelligence systems. This bill is currently pending in Assembly Judiciary Committee.
ARGUMENTS IN SUPPORT: Secure Justice writes:
In this era of disinformation and distrust in our civic institutions, we believe such efforts like SB 896 will guide agencies working to address concerns arising from the use of AI.
REGISTERED SUPPORT / OPPOSITION:
Support
Oakland Privacy
Secure Justice
Opposition
None on file.
Analysis Prepared by: Slater Sharp / P. & C.P. / (916) 319-2200
Cade Metz, Cecilia Kang, Sheera Frenkel, Stuart A. Thompson and Nico Grant, “How Tech Giants Cut Corners to Harvarst Data for A.I.,” New York Times, Apr. 6, 2024, https://www.nytimes.com/2024/04/06/technology/tech-giants-harvest-data-artificial-intelligence.html. ↑
David Thiel, “Identifying and Eliminating CSAM in Generative ML Training Data and Models,” Stanford Internet Observatory, Dec. 23, 2023. ↑
Future of Life Institute, “Asilomar AI Principles,” Aug. 11, 2017, https://futureoflife.org/open-letter/ai-principles/. ↑
National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework,” Jan. 2023, https://doi.org/10.6028/NIST.AI.100-1. ↑
The White House, “Blueprint for an AI Bill of Rights,” Oct. 2023, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. ↑
TechEquity, “AI Policy Principles,” Mar. 2024, https://techequitycollaborative.org/wp-content/uploads/2024/03/AI_Policy_Principles.pdf. ↑
Governor Gavin Newson, Executive Order N-12-23, https://www.gov.ca.gov/2023/09/06/governor-newsom-signs-executive-order-to-prepare-california-for-the-progress-of-artificial-intelligence/. ↑
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 896 |
THIRD READING
Bill No: SB 896
Author: Dodd (D), et al.
Amended: 4/10/24
Vote: 21
SENATE GOVERNMENTAL ORG. COMMITTEE: 14-0, 4/9/24
AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Jones, Nguyen, Ochoa Bogh, Padilla, Portantino, Rubio, Seyarto, Smallwood-Cuevas
NO VOTE RECORDED: Ashby, Roth
SENATE JUDICIARY COMMITTEE: 11-0, 4/16/24
AYES: Umberg, Wilk, Allen, Ashby, Caballero, Durazo, Laird, Min, Niello, Stern, Wahab
SENATE APPROPRIATIONS COMMITTEE: 7-0, 5/16/24
AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab
SUBJECT: Artificial Intelligence Accountability Act
SOURCE: Author
DIGEST:
This bill, the Artificial Intelligence (AI) Accountability Act, establishes guidelines for the use and regulation of generative AI (GenAI) within state government operations, as specified.
ANALYSIS:
Existing law:
- Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the California Department of Technology (CDT).
- Requires CDT to identify, assess, and prioritize high-risk, critical IT services and systems across state government, as determined by CDT, for modernization, stabilization, or remediation.
This bill:
- Requires the Government Operations Agency (GovOps), CDT, and the Office of Data and Innovation (ODI) to produce a State of California Benefits and Risk of GenAI Report, as specified; and to update the report, as needed, to respond to significant developments, as specified.
- Requires the Director of the Office of Emergency Services (OES), the California Cybersecurity Integration Center (Cal-CSIC), and the State Threat Assessment Center (STAC), as often as is deemed appropriate, to perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies, as specified.
- Requires GovOps, the Department of General Services (DGS), CDT, and Cal-CSIC to develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services, as specified.
- Requires GovOps to engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines, as specified.
- Requires GovOps, CDT, and ODI to develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses, as specified.
- Requires GovOps, DGS, and CDT to update, as needed, the state’s project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to the above provisions.
- Requires all state agencies and departments, in order to assist GovOps and CDT in their efforts to perform any periodic review and update per this bill, and as requested by GovOps or CDT, to conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to CDT.
- Requires a state agency or department to appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described above to CDT within 60 days of issuance of a request for an inventory.
- Requires any state agency or department to consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with GovOps, DGS, and CDT’s guidelines for public sector GenAI procurement.
- Requires CDT to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including CDT-approved environments to test those pilot projects, as specified.
- Requires GovOps, DGS, CDT, ODI, and Cal-CSIC to engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this bill.
- Requires a state agency or department to support the state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy, as specified.
- Requires legal counsel for any state agency or department to consider any potential impact of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state agency or department is being communicated through GenAI.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person to provide on the state agency’s or department’s internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
- Includes various definitions and related legislative findings a declarations.
Background
Author Statement. According to the author’s office, “California must commit to harnessing the transformative power of AI and GenAI to enhance the lives of Californians and improve the efficiency and efficacy of our state government operations. At a time when technological advancements are rapidly redefining the boundaries of possibility, it is important that we approach the integration of these powerful tools with an eye towards fairness, transparency, privacy, and accountability, while balancing the need to remain a world leader in innovation – both in the private and public sector. This bill is a first step towards ensuring that as we embrace the future, we do so in a way that upholds our key values.”
Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.
Executive Order N-12-23. Noting California’s role as a global hub for AI and the natural leader in this emerging field of technology, Governor Newsom issued Executive Order (EO) N-12-23 in September of 2023. In announcing the EO, Governor Newsom identified the need to deploy GenAI ethically and responsibly throughout state government, protect and prepare for potential harms, and remain the world’s AI leader. The EO can generally be broken up into the following elements:
- Risk-Analysis Report: Directs state agencies and departments to perform a joint-risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights (Blueprint) and the National Institute for Science and Technology’s (NIST) AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Beneficial Uses of Generative AI Report: Directs state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
- Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including CDT approved environments or “sandboxes” to test such projects.
- Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
AI Accountability Act. This bill is modeled largely after Governor Newsom’s EO N-12-23 and includes a key provision of the White House’s Blueprint for an AI Bill of Rights. Specifically, this bill requires GovOps, CDT, and ODI to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report to include an examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state and an explanation of the potential risks of the uses and the risks from bad actors and insufficiently guarding governmental systems. This bill requires those state entities to update the report, as needed, to respond to significant developments. The first Benefits and Risks of GenAI Report was the first publically released deliverable from Governor Newsom’s EO and was published in November of 2023.
Further, this bill requires, as often as deemed appropriate OES, Cal-CSIC, and the STAC to perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. That joint analysis will be provided to the Governor, and, if appropriate, public recommendations will be made reflecting changes to AI technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
This bill requires GovOps, DGS, CDT, and Cal-CSIC to develop, maintain, and periodically evaluate and revise general guidelines for the public sector procurement, uses, and required trainings for the use of GenAI, including high-risk scenarios, and including consequential decisions affecting access to essential goods and services. As required by the EO, the first edition of the State of California GenAI Guidelines for Public Sector Procurement, Uses and Training was released in March of this year.
The guidelines are available online and provide best practices and parameters to safely and effectively use this transformative technology to improve services for all Californians. This document builds on last year’s benefits and risks report and serves as interim guidance as California seeks to public the final procurement and training policy in 2025. Additionally, this bill requires GovOps, CDT, and ODI to develop, maintain, and periodically revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities.
This bill requires CDT to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including CDT-approved environments to test those pilot projects. As directed by the EO, state agencies have been working to set up these pilot projects. On May 9, 2024, GovOps announced that the state is “entering into agreements with five vendors to test, iterate and evaluate GenAI through proofs of concept (POCs). Vendors will utilize GenAI technologies developed by OpenAI, Anthropic, Google, Meta, and others to develop these solutions. The cloud environments used will be Amazon (AWS) Cloud, Google Cloud, Microsoft Azure and ServiceNow (SaaS). The state will pay each proof-of-concept vendor $1 to test in the ‘sandbox’, a secure and isolated testing environment. This gives the state and the vendors time to see what works, what doesn’t, and make necessary adjustments during a 6-month window.”
Additionally, this bill requires state agencies and departments to support the state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy. Specifically, this bill requires GovOps and CDT to make available trainings for state government worker use of state-approved GenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of GenAI while enforcing public privacy and applicable state laws and policies. This bill requires state entities to collaborate with organizations that represent state government employees and industry experts in developing and providing training.
Finally, borrowing from the White House’s Blueprint, this bill requires any state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state is being communicated through AI. Any state entity utilizes such a technology must provide on that entity’s internet website clear instructions informing the person how to directly communicate with a person from the state agency or department.
Related/Prior Legislation
SB 892 (Padilla, 2024) requires CDT to develop and adopt regulations to create an AI risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an automated decision system unless the contract includes a clause that provides a complete risk assessment, as specified. (Pending on the Senate Floor)
SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, this bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. (Pending on the Senate Floor)
SB 313 (Dodd, 2023) would have established the Office of Artificial Intelligence in state government and require state agencies to disclose when they are using generative AI to communicate with a person and to provide them an option to speak with a natural person at the agency. (Held on the Senate Appropriations Committee Suspense File)
SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights,” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
Unknown, significant one-time and ongoing costs ranging in the hundreds of thousands to millions of dollars to GovOps and CDT for staff resources to produce and regularly update the required report and provide guidance to other state agencies on GenAI use and risks (General Fund and special funds).
Unknown, significant costs ranging in the millions of dollars to OES to perform ongoing risk analysis of GenAI threats on critical infrastructure systems.
Unknown, significant costs ranging in the hundreds of thousands to millions of dollars for DGS to finalize and update state contracting procedures and guidance, conduct general supplier outreach, and complete IT work (General Fund).
Unknown, potentially significant fiscal impact ranging in the hundreds of thousands to millions of dollars across all state agencies that incorporate AI and GenAI in their operations.
SUPPORT: (Verified 5/16/2024)
Secure Justice
OPPOSITION: (Verified 5/16/2024)
None received
ARGUMENTS IN SUPPORT: In support of the bill, Secure Justice writes that, “[i]n this era of disinformation and distrust in our civic institutions, we believe such efforts like SB 896 will at minimum mitigate, if not outright eliminate, many of the known concerns arising from the use of AI.”
Prepared by: Brian Duke / G.O. / (916) 651-1530
5/17/24 10:09:09
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
SB 896 (Dodd) - Artificial Intelligence Accountability Act
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
SB 896 establishes the Artificial Intelligence Accountability Act, which tasks specified state entities with developing and periodically evaluating general guidelines for the use, regulation, and identification of potential risks of incorporating artificial intelligence (AI) and generative AI (GenAI) within state government operations.
*********** ANALYSIS ADDENDUM – SUSPENSE FILE ***********
The following information is revised to reflect amendments
adopted by the committee on May 16, 2024
Fiscal Impact:
- Unknown, significant one-time and ongoing costs ranging in the hundreds of thousands to millions of dollars to the Government Operations Agency and the California Department of Technology for staff resources to produce and regularly update the required report and provide guidance to other state agencies on GenAI use and risks (General Fund and special funds).
- Unknown, significant costs ranging in the millions of dollars to the Office of Emergency Services to perform ongoing risk analyses of GenAI threats on critical infrastructure systems.
- Unknown, significant costs ranging in the hundreds of thousands to millions of dollars for Department of General Services to finalize and update state contracting procedures and guidance, conduct general supplier outreach, and complete IT work (General Fund).
- Unknown, potentially significant fiscal impact ranging in the hundreds of thousands to millions of dollars across all state agencies that incorporate AI and GenAI in their operations.
Author Amendments:
Strike requirements pertaining to automated decision systems (ADS) and associated ADS definitions.
-- END --
SENATE COMMITTEE ON APPROPRIATIONS
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
SB 896 (Dodd) - Artificial Intelligence Accountability Act
Version: April 10, 2024 | Policy Vote: G.O. 14 - 0, JUD. 11 - 0 |
Urgency: No | Mandate: No |
Hearing Date: April 29, 2024 | Consultant: Janelle Miyashiro |
Bill Summary:
SB 896 establishes the Artificial Intelligence Accountability Act, which tasks specified state entities with developing and periodically evaluating general guidelines for the use, regulation, and identification of potential risks of incorporating artificial intelligence (AI) and generative AI (GenAI) within state government operations.
Fiscal Impact:
- Unknown, significant costs ranging into the millions to tens of millions of dollars for the Government Operations Agency (GovOps), the California Department of Technology (CDT), the Office of Data and Innovation (ODI), and the Office of Emergency Services (OES) to comply with the mandates of this bill.
- Department of General Services (DGS) anticipates:
- Total one-time cost of approximately $200,000 to finalize and publish policies, procedures, and forms and for general supplier outreach activities to promote and implement specific state contract training.
- Total ongoing costs of approximately $1,551,000 for additional staff resources to address additional IT and administrative workload, including developing policy, completing FI$Cal and State Contracting Manual updates, and providing support to DGS’s eProcurement and Business Intelligence Strategies Section (General Fund).
- Unknown, potentially significant fiscal impact ranging in the hundreds of thousands to millions of dollars across all impacted state agencies. For example, the Air Resources Board reports total costs of approximately $2,236,000 in Fiscal Year 2024-25 and $2,225,000 in FY 2025-26 and ongoing to comply with the mandates of this bill (Air Pollution Control Fund). While not all impacted state agencies will require the same level of resources, the fiscal and workload impact to these entities may still be significant.
Background:
Generally, AI is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises. Some of these most recent efforts by government and other business entities include:
- The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People (Blueprint). Released in October of 2022 by the White House’s Office of Science and Technology Policy, the Blueprint identifies five principles that should guide the design, use, and deployment of automated decision systems (ADS) and is intended to be a guide to protect all people from AI-related threats.
- National Institute of Standards and Technology’s (NIST) AI Risk Management Framework (AI RMF 1.0). Released in January of 2023, the AI RMF 1.0 is intended for voluntary use by organizations and individuals (known as AI actors) for the responsible design, development, deployment, and use of AI systems over time. The AI RMF 1.0 is divided into two parts. Part 1 discusses how organizations can frame the risks related to AI, how AI risk and trustworthiness are analyzed, and the characteristics of trustworthy AI systems. Part 2 describes four specific functions to help organizations address the risks of AI systems in practice.
- Executive Order (EO) N-12-23. Governor Gavin Newsom issued EO N-12-23 in September of 2023 “to study the development, use, and risks of AI technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government.” The EO can generally be broken up into the following elements:
- Risk-Analysis Report: Directs state agencies and departments to perform a joint-risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, directs specified agencies to issue general guidelines for public sector procurement, uses, and required training for application of GenAI—building on the Office of Science and Technology Policy’s Blueprint and NIST’s AI RMF 1.0. Directs state agencies and departments to consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Beneficial Uses of Generative AI Report: Directs state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. Requires the report to also explain the potential harms and risks for communities, government, and state government workers.
- Deployment and Analysis Framework: Directs specified state agencies to develop guidelines for other agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. Requires CDT to establish the infrastructure needed to conduct pilots of GenAI projects, including CDT approved environments or “sandboxes” to test such projects.
- Evaluation of AI Impacts on an Ongoing Basis: Directs specified state agencies and departments to periodically evaluate the potential impacts of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
Proposed Law:
- Establishes the Artificial Intelligence Accountability Act.
- Requires GovOps, CDT, and ODI to produce and update, as needed, a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes:
- An examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state.
- An explanation of the potential risks of GenAI usage to individuals, communities, and government workers with a focus on high-risk uses, including the use of GenAI to make a consequential decision affecting access to essential goods and services.
- An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
- Requires, as deemed appropriate by the Director of OES, requires the California Cybersecurity Integration Center (Cal-CSIC) and the State Threat Assessment Center to perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy and other critical infrastructure.
- Requires the analysis to be provided to the Governor, and if appropriate, requires public recommendations be made reflecting changes in AI technology, its applications, and risk management.
- Requires GovOps, DGS, CDT, and Cal-CSIC to develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI.
- Requires the guideline to build on guidance from the Blueprint and AI RMF 1.0, or succeeding documents. Requires the guideline to address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by GenAI.
- Requires GovOps, CDT, and ODI to develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
- Requires the guidelines to inform whether and how a state agency or department deploys a particular GenAI tool.
- Requires GovOps, DGS, and CDT to update, as needed, the state’s project approval, procurement, and contract terms to incorporate analysis and feedback on the use and adoption of GenAI tools.
- Requires state agencies and departments, as requested by GovOps and CDT, to conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to CDT, and tasks CDT with administering the inventory.
- Requires a state agency or department to appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory to CDT within 60 days of GovOps or CDT request.
- Requires a state agency or department to consider GenAI procurement and enterprise use opportunities consistent with GovOps, DGS, and CDT guidelines for public sector GenAI procurement to improve efficiency, effectiveness, accessibility, and equity of government operations.
- Requires CDT to establish and maintain the infrastructure to conduct pilot projects of GenAI projects.
- Requires GovOps, DGS, CDT, ODI, and Cal-CSIC to engage with the Legislature and relevant stakeholders in the development and revision of any guidelines, criteria, reports, or training.
- Requires a state agency or department to support the state government workforce to prepare for the next generation of skills needed to thrive in the GenAI economy.
- Requires GovOps, CDT, and any other agencies deemed necessary to make trainings available for state government worker use of state-approved GenAI tools. If appropriate, requires CDT and any other agency or department deemed necessary to collaborate with organizations that represent state government employees and industry experts on developing and providing training.
- Requires GovOps, in consultation with appropriate state agencies and organizations that represent state government employees, to establish criteria to evaluate the impact of GenAI on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
- Requires legal counsel for any state agency or department to consider any potential impact of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state agency or department is being communicated through AI.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person shall provide on the state agency’s or department’s internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
- Requires a state agency or department to evaluate ADSs for potential risks before adoption.
- Requires a high-risk ADS to receive appropriate consultation, testing, risk identification, and mitigation.
- Prohibits adoption or utilization of a high-risk ADS without prior approval of the director of that state agency or department or that person’s designee.
- Requires a high-risk ADS to receive ongoing monitoring and clear organizational oversight.
- Establishes definitions and states legislative findings and declarations.
Related Legislation:
SB 892 (Padilla) requires CDT to develop and adopt regulations for an AI risk management standard regarding state procurement and use of automated decision systems (ADS), as specified. SB 892 is pending in this committee.
SB 893 (Padilla, 2024) establishes the California Artificial Intelligence Research Hub in the Government Operations Agency to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development, as specified. SB 893 is pending in this committee.
SB 942 (Becker, 2024) the California AI Transparency Act, among other things, requires a covered provider to create an AI detection tool by which a person can query the covered provide as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a GenAI system, as specified. SB 942 is pending in this committee.
SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, SB 1047 establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. SB 1047 is pending in this committee.
Staff Comments:
A breakdown of DGS’s estimated ongoing staff costs are as follows:
Staff Classification | Positions Required | Total Annual |
IT Associate | 1.0 | $167,000 |
IT Specialist I | 1.0 | $201,000 |
IT Specialist II | 4.0 | $932,000 |
IT Supervisor | 1.0 | $251,000 |
$1,551,000 |
-- END --
Senator Thomas Umberg, Chair
2023-2024 Regular Session
Bill No: SB 896 (Author:Dodd)
Version: April 10, 2024
Hearing Date: April 16, 2024
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
Artificial Intelligence Accountability Act
DIGEST
This bill largely codifies Governor Newsom’s executive order on the use of Generative artificial intelligence (GenAI). The bill requires assessments of the beneficial uses, potential harms, and risks to critical infrastructure of GenAI. The bill calls for the development of guidelines for public sector procurement, uses, and required trainings for the use of GenAI. The bill places obligations on state entities with respect to the use of GenAI and automated decisionmaking systems.
EXECUTIVE SUMMARY
The transformative power of AI and specifically GenAI is unquestionable and it offers numerous benefits for society, including state government. It can enhance efficiency and effectiveness in various sectors freeing up human resources. AI can improve decision-making processes by analyzing vast amounts of data to identify patterns, trends, and potential insights. This capability enables state governments to make more informed policy decisions, allocate resources more efficiently, and enhance public services. However, as with most technologies, there are also inherent risks and challenges. One major concern is the potential for bias in AI algorithms, which could result in discriminatory outcomes, particularly in areas such as law enforcement, healthcare, and social services. Furthermore, the reliance on AI systems may lead to job displacement, as automation replaces certain roles previously performed by humans. Ensuring transparency, accountability, and fairness in AI systems is crucial to mitigate these risks and maintain public trust. Additionally, there are concerns about data privacy and security, as GenAI systems rely on vast amounts of sensitive information.
This bill largely codifies Governor Newsom’s executive order on GenAI, requiring assessments of the uses, potential harms, and risks of GenAI. The bill calls for the development of guidelines for public sector procurement, uses, and required trainings for the use of GenAI. The bill places obligations on state entities with respect to the use of GenAI and automated decisionmaking systems (ADS). This bill is author-sponsored. It is supported by Secure Justice. No timely opposition has been received by the Committee. The bill passed out of the Senate Governmental Organization Committee on a vote of 14 to 0.
PROPOSED CHANGES TO THE LAW
Existing law:
- Establishes the California Department of Technology (CDT) within the California Government Operations Agency (GovOps), under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Gov. Code Sec. 11545(a).)
- Provides that the duties of the Director include:
- advising the Governor on the strategic management and direction of the state’s information technology (IT) resources;
- establishing and enforcing state IT strategic plans, policies, standards, and enterprise architecture, as specified;
- minimizing overlap, redundancy, and cost in state IT operations by promoting the efficient and effective use of information technology;
- providing technology direction to agency and department chief information officers to ensure the integration of statewide technology initiatives, compliance with IT policies and standards, and the promotion of the alignment and effective management of IT services; and
- working to improve organizational maturity and capacity in the effective management of IT; and establishing performance management and improvement processes to ensure state IT systems and services are efficient and effective. (Gov. Code § 11545(b).)
- Expresses the intent of the Legislature that policies and procedures developed by CDT and Department of General Services (DGS) pertaining to the acquisition of IT goods and services provide for all of the following: the expeditious and value-effective acquisition of IT goods and services to satisfy state requirements; the acquisition of IT goods and services within a competitive framework; the delegation of authority by DGS to each state agency that has demonstrated to DGS’s satisfaction the ability to conduct value-effective IT goods and services acquisitions; and the review and resolution of protests submitted by any bidders with respect to any IT goods and services acquisitions. (Pub. Con. Code § 12101.)
- Requires CDT, on or before September 1, 2024, to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. (Gov. Code § 11546.45.5(b).)
- Requires the comprehensive inventory to include a description of all of the following:
- any decision the ADS can make or support, the intended benefits of that use, and the alternatives to that use;
- the results of any research assessing the efficacy and relative benefits of the uses and alternatives of the ADS described above;
- the categories of data and personal information the ADS uses to make its decisions; and
- the measures in place, if any, to mitigate the risks, including cybersecurity risk and the risk of inaccurate, unfairly discriminatory, or biased decisions, of the ADS, including performance metrics, cybersecurity controls, privacy controls, risk assessments or audits for potential risks, and measures or processes in place to contest an automated decision. (Gov. Code § 11546.45.5(c).)
- Requires CDT, on or before January 1, 2025, and annually thereafter, to submit a report, as specified, of the comprehensive inventory to the Assembly Committee on Privacy and Consumer Protection and the Senate Committee on Governmental Organization. This requirement expires on January 1, 2029. (Gov. Code § 11546.45.5(d).)
- Defines the following terms:
- “Automated decision system” means a computational process derived from machine learning, statistical modeling, data analytics, or AI that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. ADS does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
- “High-risk automated decision system” means an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.
- “State agency” includes every state office, department, division, bureau, the California State University, the Board of Parole Hearings, and specified boards. It does not include the University of California, the Legislature, the judicial branch, or any board, except as provided. (Gov. Code § 11546.45.5(a).)
This bill:
- Establishes the Artificial Intelligence Accountability Act.
- Defines the relevant terms, including:
- “Artificial intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
- “Automated decision system” means a computational process derived from machine learning, statistical modeling, data analytics, or AI that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. It does not include a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
- “Generative artificial intelligence” means the class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
- “High-risk automated decision system” means an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.
- Requires GovOps, the Office of Data and Innovation (ODI), and CDT to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report that includes all of the following:
- An examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state.
- An explanation of the potential risks of the uses to individuals, communities, and government workers with a focus on high-risk uses, including the use of GenAI to make a consequential decision affecting access to essential goods and services.
- An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
- Requires the above entities to update the report, as needed, to respond to significant developments and, as appropriate, consult with academia, industry experts, and organizations that represent state government employees. These entities are also required to develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services, as specified. These shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by GenAI. Guidelines must also be produced for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
- Requires GovOPS, ODI, and CDT to consult with a broad range of experts and stakeholders. These processes shall inform updates to the state’s project approval, procurement, and contract terms.
- Requires all state agencies and departments, as requested by GovOps or CDT, to conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to the CDT, which shall administer the inventory, as specified.
- Provides that the Director of the Office of Emergency Services (OES), the California Cybersecurity Integration Center (Cal-CSIC), and the State Threat Assessment Center (STAC) shall, as often as is deemed appropriate by them, perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. They are also required to develop a strategy, in consultation with outside experts, to assess similar potential threats to other critical infrastructure.
- Requires the above analysis to be provided to the Governor and public recommendations are to be made reflecting changes to AI technology, its applications, and risk management, including further actions to guard against potential threats and vulnerabilities.
- Requires state agencies and departments to consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with the guidelines for public sector GenAI procurement. CDT shall establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including CDT-approved environments to test those pilot projects, as specified. Such pilot projects are required to measure:
- How GenAI can improve Californians’ experience with, and access to, government services.
- How GenAI can support state employees in the performance of their duties in addition to any domain-specific impacts to be measured by the state agency or department.
- Requires GovOps, ODI, and CDT to engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant hereto. These entities, and any other agencies deemed necessary, shall make available trainings for state government worker use of state-approved GenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of GenAI, while enforcing public privacy and applicable state laws and policies, as provided.
- Requires GovOps, in consultation with appropriate state agencies and organizations that represent state government employees, to establish criteria to evaluate the impact of GenAI on the state government workforce and provide guidelines on how state agencies and departments can support state government employees to use these tools effectively and respond to these technological advancements.
- Directs legal counsel for any state agency or department to consider any potential impact of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state agency or department is being communicated through AI. They must also provide on the state agency’s or department’s website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
- Requires that an automated decisionmaking system utilized by a state agency or department to be evaluated for risk potential before adoption. A highrisk automated decision system shall receive appropriate consultation, testing, risk identification, and mitigation consistent hereto and shall not be adopted or utilized without prior approval of the director of a state agency or department or that person’s designee before being adopted and utilized by a state agency or department. Such a highrisk system shall receive ongoing monitoring and clear organizational oversight.
COMMENTS
- Frameworks for responsible development and accountability in AI
Owing to recent advances in processing power and the rise of big data, AI’s capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact, and consume media. Since the widespread introduction of AI systems such as ChatGPT, the world has been in awe of the powers of GenAI, which is a type of artificial intelligence that can create new content, such as text, images, code, or music, by learning from existing data. GenAI models can produce realistic and novel artifacts that resemble the data they were trained on, but do not copy it. For example, generative AI can write a poem, draw a picture, or compose a song based on a given prompt or theme.
In ways we may not fully comprehend, AI empowers and encumbers us. It has been used to accelerate productivity, achieve efficiencies, liberate us from drudgery, write our college essay, help us understand and enjoy the world, connect with each other, and live longer, fuller lives. It has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy. For good or ill, its transformative potential seems boundless.
With these recent dramatic advances in the capabilities of AI systems, the need for frameworks for accountability and responsible development have become ever more urgent.
In January of 2017, AI researchers, economists, legal scholars, ethicists, and philosophers met in Asilomar, California to discuss principles for managing the responsible development of AI. The collaboration resulted in the Asilomar Principles. Aspirational rather than prescriptive, these 23 principles were intended to initiate and frame a dialogue by providing direction and guidance for policymakers, researchers, and developers. Its endorsers include 1,200 leading experts in the field of AI, including DeepMind founder Demis Hassabis and the late Stephen Hawking.
The Legislature subsequently adopted ACR 215 (Kiley, Ch. 206, Stats. 2018), which added the State of California to that list by endorsing the Asilomar Principles as guiding values for the development of artificial intelligence and related public policy. In broad strokes, those principles aim to do the following:
- Research issues: create beneficial AI; direct funding toward beneficial innovation; maintain constructive and healthy exchanges between AI researchers and policymakers; promote a culture of trust, cooperation, and transparency among researchers and developers of AI; and avoid corner-cutting on safety standards.
- Ethics and values: promote safety, failure transparency, judicial transparency, and responsible innovation; align human values with innovation; protect privacy and liberty; ensure that the benefits and prosperity created by AI are broadly shared; maintain human control over AI; develop AI that supports rather than subverts social and civil processes; and avoid an AI arms race.
- Longer-term issues: avoid assumptions regarding the capabilities of AI; give AI its due attention; and recognize that its risks are potentially catastrophic or existential. [emphasis added]
As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.
While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.
More recently the Biden Administration has published its Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the
design, use, and deployment of AI to protect the rights of the American public:
- Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
- Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
- Data Privacy: You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used. You should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used. Systems should not employ user experience and design decisions that obfuscate user choice or burden users with defaults that are privacy invasive. Consent should only be used to justify collection of data in cases where it can be appropriately and meaningfully given. Any consent requests should be brief, be understandable in plain language, and give you agency over data collection and the specific context of use; current hard-to-understand notice-and-choice practices for broad uses of data should be changed. Enhanced protections and restrictions for data and inferences related to sensitive domains, including health, work, education, criminal justice, and finance, and for data pertaining to youth should put you first. In sensitive domains, your data and related inferences should only be used for necessary functions, and you should be protected by ethical review and use prohibitions. You and your communities should be free from unchecked surveillance; surveillance technologies should be subject to heightened oversight that includes at least pre-deployment assessment of their potential harms and scope limits to protect privacy and civil liberties. Continuous surveillance and monitoring should not be used in education, work, housing, or in other contexts where the use of such surveillance technologies is likely to limit rights, opportunities, or access. Whenever possible, you should have access to reporting that confirms your data decisions have been respected and provides an assessment of the potential impact of surveillance technologies on your rights, opportunities, or access.
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.[1]
TechEquity, an organization committed to ensuring technology’s evolution benefits everyone equitably, has also laid out their straightforward AI Policy Principles:
- People who are impacted by AI must have agency to shape the technology that dictates their access to critical needs like employment, housing, and healthcare.
- The burden of proof must lie with developers, vendors, and deployers to demonstrate that their tools do not create harm—and regulators, as well as private [individuals], should be empowered to hold them accountable.
- Concentrated power and information asymmetries must be addressed in order to effectively regulate the technology.
- A framework for California: GenAI uses, risks, and benefits
Seeking to establish a framework for California, Governor Gavin Newsom issued Executive Order N-12-23 “to study the development, use, and risks of artificial intelligence (AI) technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government.”[2]
The executive order includes the following provisions:
- Risk-Analysis Report: Directs state agencies and departments to perform a joint risk-analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Beneficial Uses of GenAI Report: Direct state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
- Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including CDT approved environments or “sandboxes” to test such projects.
- State Employee Training: To support California’s state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy, agencies will provide trainings for state government workers to use state-approved GenAI to achieve equitable outcomes, and will establish criteria to evaluate the impact of GenAI to the state government workforce.
- Legislative Engagement: Engage with legislative partners and key stakeholders, including academic institutions, in a formal process to develop policy recommendations for responsible use of AI, including any guidelines, criteria, reports, and/or training.
- Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
This bill establishes the Artificial Intelligence Accountability Act. The bill codifies most of the key provisions of Executive Order N-12-23. According to the author:
The AI Accountability Act ensures state agencies advance safeguards and consumer protections around AI by incorporating and building upon recent directives from President Joe Biden and Governor Gavin Newsom. SB 896 encourages continued innovation while ensuring that the rights and opportunities of all Californians are protected. When used ethically and transparently, AI has the potential to dramatically improve service delivery outcomes and increase access to and utilization of government programs. However, just as humans have explicit and implicit biases, AI has the capacity to act as a mirror by reflecting and amplifying these biases. SB 896 builds on previous federal and state executive orders by guiding the decision-making of state agencies, departments, and subdivisions in the review, adoption, management, governance, and regulation of automated decision-making systems and generative AI.
Specifically, this bill requires GovOps, CDT, and ODI to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report to include an examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state and an explanation of the potential risks of the uses and the risks from bad actors and insufficiently guarding governmental systems.
The bill directs OES, Cal-CSIC, and STAC to perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. The bill calls for the development of guidelines for public sector procurement, uses, and required trainings for the use of GenAI. This includes an assessment of the impact on vulnerable communities. The bill calls for the development of pilot projects, something already underway in response to the Governor’s executive order. Additionally, this bill requires state agencies and departments to support the state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy.
Following principles laid out in the previous section, including the White House’s Blueprint, the bill requires any state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state is being communicated through AI. Any state entity that utilizes such a technology must provide on that entity’s internet website clear instructions informing the person how to directly communicate with a person from the state agency or department. This ensures Californians are on notice of when GenAI is in use and are provided a human alternative.
The bill also places obligations on state entities with respect to the use of GenAI and automated decisionmaking systems (ADS). ADS are algorithm-driven applications that can assist or supplant human decisionmaking processes in areas such as credit decisions, employment screening, insurance eligibility, and the delivery of government services. ADS process enormous datasets and make decisions with speed and reliability that vastly exceed human capabilities. However, poorly designed or poorly understood systems can create unfair, biased, and inaccurate results. When deployed by government agencies, flawed ADS may disproportionately harm low-income families and communities of color and undermine trust in the public sector. Moreover, norms of participatory governance and due process may be jeopardized when ADS affects agency policymaking, adjudications, or enforcement.
Last year, AB 302 (Ward, Ch. 800, Stats. 2023) required CDT, on or before September 1, 2024, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. This bill requires that ADS utilized by a state agency or department be evaluated for risk potential before adoption. High-risk ADS must receive appropriate consultation, testing, risk identification, and mitigation and shall not be adopted or utilized without prior approval of the director of a state agency or department or that person’s designee before being adopted and utilized by a state agency or department. “High-risk ADS” is defined as ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. A high-risk ADS that is utilized by a state agency or department is required to receive ongoing monitoring and clear organizational oversight.
- Stakeholder positions
Writing in support, Secure Justice states: “In this era of disinformation and distrust in our civic institutions, we believe such efforts like SB 896 will at minimum mitigate, if not outright eliminate, many of the known concerns arising from the use of AI.”
SUPPORT
Secure Justice
OPPOSITION
None received
RELATED LEGISLATION
Pending Legislation:
SB 892 (Padilla, 2024) directs CDT to establish an AI risk management standard, modeled after mainstream, widely accepted publications, to guide the procurement, use, and oversight of ADS in government agencies. Agencies are required to include specific clauses in contracts for the procurement of ADS after CDT promulgates attendant regulations. SB 892 is currently in this Committee.
SB 893 (Padilla, 2024) requires GovOps, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California Artificial Intelligence Research Hub in GovOps, as prescribed. SB 893 requires the hub to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development that seeks to harness the technology’s full potential for public benefit while safeguarding privacy, advancing security, and addressing risks and potential harms to society, as prescribed. SB 893 is currently in this Committee.
SB 942 (Becker, 2024) establishes the California AI Transparency Act, which, among other things, requires a covered provider, as defined, to create an AI detection tool by which a person can query the covered provider as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a generative AI system, as defined, provided by the covered provider that meets certain criteria. Covered providers are required to include in AI-generated content a visible disclosure that, among other things, includes a clear and conspicuous notice that identifies the content as generated by AI. SB 942 requires a covered provider to register with CDT and provide them a URL to any AI detection tool it has created. SB 942 is currently in this Committee.
SCR 17 (Dodd, 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems. SCR 17 is currently in the Assembly Privacy and Consumer Protection Committee.
AB 331 (Bauer-Kahan, 2023) prohibits “algorithmic discrimination,” that is, use of an automated decision tool to contribute to unjustified differential treatment or outcomes that may have a significant effect on a person’s life. It requires any deployer of an automated decision tool to perform an impact assessment for those tools and to notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision. AB 331 was held in the Senate Appropriations Committee.
AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. AB 2013 is currently in the Assembly Privacy and Consumer Protection Committee.
AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer and a developer of an automated decision tool to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The assessments must be provided to the Civil Rights Department within 7 days of a request. AB 2930 requires a deployer to, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. AB 2930 is currently in the Assembly Privacy and Consumer Protection Committee.
Prior Legislation:
AB 302 (Ward, Ch. 800, Stats. 2023) See Comment 2.
AB 13 (Chau, 2021) would have established the Automated Decision Systems Accountability Act, which, in the context of the State’s procurement policies, would have promoted oversight over ADS that pose a high risk of adverse impacts on individual rights. The bill was eventually gutted and amended to address a different topic.
SB 444 (Umberg, 2019) would have requested the Regents of the University of California (UC) to enact a resolution authorizing the law schools at UC Berkeley and UC Irvine to participate in a pilot project to develop AI or machine-learning solutions to address access to justice issues faced by self-representing litigants in their respective courts. The bill died in the Assembly Higher Education Committee.
AB 1576 (Calderon, 2019) would have required the Secretary GovOps to appoint participants to an AI working group to evaluate the uses, risks, benefits, and legal implications associated with the development and deployment of AI by California-based businesses. The bill was held on the Senate Appropriations Committee suspense file.
SJR 6 (Chang, Res. Ch. 112, Stats. 2019) urged the President and the Congress of the United States to develop a comprehensive AI Advisory Committee and to adopt a comprehensive AI policy.
ACR 215 (Kiley, Resolution Ch. 206, Stats. 2018) See Comment 1.
PRIOR VOTES:
Senate Governmental Organization Committee (Ayes 14, Noes 0)
**************
Blueprint For An AI Bill Of Rights (October 2022) Office of Science and Technology Policy, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. All internet citations are current as of April 10, 2024. ↑
Press Release, Governor Newsom Signs Executive Order to Prepare California for the Progress of Artificial Intelligence (September 6, 2023) Office of Governor Gavin Newsom, https://www.gov.ca.gov/2023/09/06/governor-newsom-signs-executive-order-to-prepare-california-for-the-progress-of-artificial-intelligence/. ↑
SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION
Senator Bill Dodd
Chair
2023 - 2024 Regular
Bill No: 896 Hearing Date: 4/9/2024
Author: | |||
Version: | 1/3/2024 Introduced | ||
Urgency: | No | Fiscal: | Yes |
Consultant: | Brian Duke | ||
SUBJECT: Artificial Intelligence Accountability Act
DIGEST: This bill, the Artificial Intelligence (AI) Accountability Act, establishes guidelines for the use and regulation of AI and generative AI (GenAI) within state government operations. Specifically, this bill requires the development and production of a Benefits and Risk of Generative Artificial Intelligence Report; requires joint risk analyses of potential threats posed by GenAI to California’s critical energy infrastructure, as specified; requires the development of guidelines for the procurement of AI and GenAI, as specified; requires the support and training of the state government workforce for the next generation of skills needed to thrive in the GenAI economy, as specified; and, requires a state agency to identify a person when that person is communicating with a state agency directly through GenAI, as specified.
ANALYSIS:
Existing law:
- Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the California Department of Technology (CDT).
- Requires CDT to identify, assess, and prioritize high-risk, critical information technology services and systems across state government, as determined by CDT, for modernization, stabilization, or remediation.
- Requires the Secretary of the Government Operations Agency (GovOps) to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital provenance, as specified.
- Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk automated decision systems (ADS) that have been proposed for use or are being used, developed, or procured by state agencies, as specified.
- Defines “automated decision system” to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons. “Automated decision system” does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
This bill:
- Requires GovOps, CDT, and the Office of Data and Innovation (ODI) to produce a State of California Benefits and Risk of GenAI Report that includes all of the following:
- An examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state.
- An explanation of the potential risks of the uses described above to individuals, communities, and government workers with a focus on high-risk uses, including the use of GenAI to make a consequential decision affecting access to essential goods and services.
- An explanation of risks from bad actors and insufficiently guarded governmental systems, unintended or emergent effects, and potential risks toward democratic and legal processes, public health and safety, and the economy.
- Requires GovOps, CDT, and ODI to update the report, as needed, to respond to significant developments and, as appropriate, consult with academia, industry experts, and organizations that represent state government employees.
- Requires the Director of the Office of Emergency Services (OES), the California Cybersecurity Integration Center (Cal-CSIC), and the State Threat Assessment Center (STAC), as often as is deemed appropriate, to perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies.
- Requires OES, Cal-CSIC, and the STAC to develop, in consultation with appropriate external experts from academia and industry, a strategy to assess similar potential threats to other critical infrastructure.
- Requires that risk analysis be provided to the Governor, and if appropriate, requires public recommendations be made reflecting changes to AI technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
- Requires GovOps, the Department of General Services (DGS), CDT, and Cal-CSIC to develop, maintain, and periodically evaluate and revise general guidelines for public sector procurement, uses, and required trainings for the use of GenAI, including for high-risk scenarios, and including for consequential decisions affecting access to essential goods and services.
- Requires the guidelines described above to be built on guidance from the White House publication titled Blueprint for an AI Bill of Rights and the National Institute of Standards and Technology’s AI Risk Management Framework and requires the guidelines to address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by GenAI.
- Requires GovOps to engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines.
- Requires GovOps, DGS, CDT, and Cal-CSIC, for the purposes of the periodic evaluation and revision required above, to evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
- Requires GovOps, CDT, and ODI to develop, maintain, and periodically evaluate and revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities, including criteria to evaluate equitable outcomes in deployment and implementation of high-risk uses.
- Requires the guidelines described above to inform whether and how a state agency or department deploys a particular GenAI tool and requires GovOps to engage and consult with organizations that represent state government employees and industry experts, including, but not limited to, trust and safety experts, academic researchers, and research institutions in developing the guidelines.
- Requires, for the purposes of the periodic evaluation and revision described above, GovOps, DGS, CDT, and Cal-CSIC to periodically evaluate any need to revise the guidelines and establish a consultative process by which to do so with academia, industry experts, and organizations that represent state government employees.
- Requires GovOps, DGS, and CDT to update, as needed, the state’s project approval, procurement, and contract terms to incorporate analysis and feedback obtained pursuant to the above provisions.
- Requires all state agencies and departments, in order to assist GovOps and CDT in their efforts to perform any periodic review and update per this bill, and as requested by GovOps or CDT, to conduct and submit an inventory of all current high-risk uses of GenAI within the agency or department to CDT.
- Requires a state agency or department to appoint a senior level management personnel who will be responsible for maintaining, conducting, and reporting the results of the inventory described above to CDT within 60 days of issuance of a request for an inventory.
- Requires any state agency or department to consider procurement and enterprise use opportunities in which GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations consistent with GovOps, DGS, and CDT’s guidelines for public sector GenAI procurement.
- Requires CDT to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including CDT-approved environments to test those pilot projects.
- Requires an environment created pursuant to this bill be made available to any state agency or department to help evaluate GenAI tools and services, to further safe, ethical, and responsible implementations, and to inform decisions to use GenAI consistent with state guidelines.
- Requires, by an unspecified date, any state agency or department to consider pilot projects of GenAI applications, in consultation with organizations that represent state government employees and appropriate experts from academia and industry.
- Requires those pilot projects to measure both of the following:
- How GenAI can improve Californians’ experience with, and access to, government services.
- How GenAI can support state employees in the performance of their duties in addition to any domain-specific impacts to be measured by the state agency or department.
- Requires GovOps, DGS, CDT, ODI, and Cal-CSIC to engage with the Legislature and relevant stakeholders, including historically vulnerable and marginalized communities and organizations that represent state government employees, in the development and revision of any guidelines, criteria, reports, or training pursuant to this bill.
- Requires a state agency or department to support the state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy by complying with both of the following:
- Requires GovOps, CDT, and any other agencies deemed necessary, to make available trainings for state government worker use of state-approved GenAI tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of GenAI, while enforcing public privacy and applicable state laws and policies.
- Requires GovOps, in consultation with appropriate state agencies and organizations that represent state government employees, to establish criteria to evaluate the impact of GenAI on the state government workforce and provide guidelines on how state agencies and departments can support state government employees.
- Requires legal counsel for any state agency or department to consider any potential impact of GenAI on regulatory issues under the respective agency’s or department’s authority and recommend necessary updates, if appropriate, as a result of this evolving technology.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state agency or department is being communicated through GenAI.
- Requires a state agency or department that utilizes GenAI to directly communicate with a person to provide on the state agency’s or department’s internet website clear instructions, or a link to a web page with clear instructions, informing the person how to directly communicate with a person from the state agency or department.
- Requires an ADS utilized by a state agency or department to be evaluated for risk potential before adoption.
- Requires a high-risk ADS to receive appropriate consultation, testing, risk identification, and mitigation consistent with this bill and prohibits that technology from being adopted or utilized without prior approval of the director of a state agency or department or that person’s designee before being adopted and utilized by a state agency or department.
- Requires a high-risk ADS that is utilized by a state agency or department to receive ongoing monitoring and clear organizational oversight.
- Defines “generative artificial intelligence” to mean the class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content, including images, videos, audio, text, and other digital content.
- Includes related legislative findings and declarations.
Background
Author Statement. According to the author’s office, “California must commit to harnessing the transformative power of AI and GenAI to enhance the lives of Californians and improve the efficiency and efficacy of our state government operations. At a time when technological advancements are rapidly redefining the boundaries of possibility, it is important that we approach the integration of these powerful tools with an eye towards fairness, transparency, privacy, and accountability, while balancing the need to remain a world leader in innovation – both in the private and public sector. This bill is a first step towards ensuring that as we embrace the future, we do so in a way that upholds our key values.”
Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.
Executive Order N-12-23. Noting California’s role as a global hub for AI and the natural leader in this emerging field of technology, Governor Newsom issued Executive Order (EO) N-12-23 in September of 2023. In announcing the EO, Governor Newsom identified the need to deploy GenAI ethically and responsibly throughout state government, protect and prepare for potential harms, and remain the world’s AI leader. The EO can generally be broken up into the following elements:
- Risk-Analysis Report: Directs state agencies and departments to perform a joint-risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s (NIST) AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Beneficial Uses of Generative AI Report: Directs state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
- Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including CDT approved environments or “sandboxes” to test such projects.
- Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
This bill seeks to codify many of the key provisions of EO N-12-23.
The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.
The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.
Secondly, the Blueprint includes “Algorithmic Discrimination Protections.” Algorithmic discrimination occurs when automated systems contribute to unjustified different treatment or impacts disfavoring people based on their race, color, ethnicity, sex (including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual orientation), religion, age, national origin, disability, veteran status, genetic information, or any other classification protected by law. Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way.
Additionally, the Blueprint discusses “Data Privacy,” stating that individuals should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used.
Further, the Blueprint identifies “Notice and Explanation.” Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
The final principle identified by the Blueprint is “Human Alternatives, Consideration, and Fallback.” The Blueprint states that you should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts. In some cases, a human or other alternative may be required by law.
National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). The National Institute of Standards and Technology (NIST) was founded in 1901 and is now part of the U.S. Department of Commerce. NIST is one of the nation’s oldest physical science laboratories. Today, NIST measurements support the smallest of technologies to the largest and most complex of human-made creations – from nanoscale devices to tiny that tens of thousands can fit on the end of a single human hair up to earthquake-resistant skyscrapers and global communication networks.
As directed by the National AI Initiative Act of 2020, NIST released the AI RMF in January 2023, to better manage risks to individuals, organizations, and society associated with AI. NIST’s AI RMF is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. The framework was developed through a consensus-driven, open, transparent, and collaborative process that included a Request for Information, several draft versions for public comments, multiple workshops, and other opportunities to provide input. It is intended to build on, align with, and support AI risk management efforts by others. The framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
- AI technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high- or low-probability, systemic or localized, and high- or low-impact.
- While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits- can emerge from an interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
- These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society.
- AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by promoting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.
Federal Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence. In October of 2023, President Biden issued a federal EO on Safe, Secure, and Trustworthy AI. The order outlines a multifaceted approach for ensuring the development and deployment of AI technologies are safe, secure, and trustworthy, while also advancing American leadership globally. Specifically, the order focuses on the following:
- New Standards for AI Safety and Security. Requires developers of powerful AI systems to share safety test results with the U.S. government. Requires NIST to establish rigorous standards for AI safety, including extensive red-team testing. Includes measures to protect against the use of AI in engineering dangerous biological materials and AI-enabled fraud and deception.
- Protecting American’s Privacy. Prioritizes the development of AI technologies that preserve privacy. Involves funding and coordination efforts to advance privacy-preserving technologies and tools.
- Advancing Equity and Civil Rights. Provides clear guidance to prevent AI algorithms from exacerbating discrimination in housing, federal benefits, and federal contracting.
- Supporting Workers. Develops principles to address job displacement, evaluate job applications fairly, and ensure workplace equity.
- Ensuring Responsible Government Use of AI. Issues guidelines for government agencies on AI procurement, deployment, and infrastructure modernization.
California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to move forward with draft ADS technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students. According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”
Further, CalMatters states that under the proposed rules, “businesses must notify people before using AI. If people opt out of interacting with an AI model, businesses cannot discriminate against people for that choice. If people agree to use an AI service or tool, businesses must respond to requests by individuals about how they use their personal information to make predictions. The rules would also require employers or third-party contractors to carry out risk assessments to evaluate the performance of their technology.”
According to the CPPA, the proposed framework includes the following three key components:
- Pre-use Notice Requirements. A business must provide information to the consumer about how it proposes to use the ADS so that the consumer can decide whether to opt-out or proceed, and whether to exercise their access right.
- Opt-out Right Requirements. For opt-out requests submitted before the business initiates the processing, the business must not process consumer’s personal information using that ADS. For opt-out requests submitted after the business has initiated the processing, the business must cease processing the consumer’s personal information using that ADS and notify relevant entities of the opt-out and instruct them to comply.
- Access Rights Requirements. This involves the purpose of using the ADS; the output with respect to the consumer; how the businesses used the output with respect to the consumer; how the ADS worked with respect to the consumer; and that the businesses cannot retaliate against consumers for exercising CPPA rights.
The CPPA will spend the rest of the spring and summer performing a statewide informational outreach tour before initiating formal rulemaking which is expected later this year.
AI Accountability Act. This bill is modeled largely after Governor Newsom’s EO N-12-23 and includes a key provision of the White House’s Blueprint. Specifically, this bill requires GovOps, CDT, and ODI to produce a State of California Benefits and Risk of Generative Artificial Intelligence Report to include an examination of the most significant, potentially beneficial uses for deployment of GenAI tools by the state and an explanation of the potential risks of the uses and the risks from bad actors and insufficiently guarding governmental systems. This bill requires those state entities to update the report, as needed, to respond to significant developments. The first Benefits and Risks of GenAI Report was the first publically released deliverable from Governor Newsom’s EO and was published in November of 2023.
Further, this bill requires, as often as deemed appropriate OES, Cal-CSIC, and the STAC to perform a joint risk analysis of potential threats posed by the use of GenAI to California’s critical energy infrastructure, including those that could lead to mass casualty events and environmental emergencies. That joint analysis will be provided to the Governor, and, if appropriate, public recommendations will be made reflecting changes to AI technology, its applications, and risk management, including further private actions, administrative actions, and collaboration with the Legislature to guard against potential threats and vulnerabilities.
This bill requires GovOps, DGS, CDT, and Cal-CSIC to develop, maintain, and periodically evaluate and revise general guidelines for the public sector procurement, uses, and required trainings for the use of GenAI, including high-risk scenarios, and including consequential decisions affecting access to essential goods and services. As required by the EO, the first edition of the State of California GenAI Guidelines for Public Sector Procurement, Uses and Training was released in March of this year.
The guidelines are available online and provide best practices and parameters to safely and effectively use this transformative technology to improve services for all Californians. This document builds on last year’s benefits and risks report and serves as interim guidance as California seeks to public the final procurement and training policy in 2025. Additionally, this bill requires GovOps, CDT, and ODI to develop, maintain, and periodically revise guidelines for state agencies and departments to analyze the impact that adopting a GenAI tool may have on vulnerable communities.
This bill requires CDT to establish and maintain the infrastructure to conduct pilot projects of GenAI projects, including CDT-approved environments to test those pilot projects. As directed by the EO, state agencies have been working to set up these pilot projects. A February 8, 2024 article by CalMatters highlighted the current five proofs of concept under the pilot project. According to the article, in addition to a current Department of Tax and Fee Administration tax advice GenAI proposal, “Caltrans has two projects to explore if GenAI can reduce traffic congestion and fatal accidents and the state’s Health and Human Services Agency has two trials underway to explore if GenAI can make it easier for people to understand and attain public benefits, and assist in health care facility inspections.”
Additionally, this bill requires state agencies and departments to support the state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy. Specifically, this bill requires GovOps and CDT to make available trainings for state government worker use of state-approved GenAI intelligence tools to achieve equitable outcomes and to identify and mitigate potential output inaccuracies, fabricated text, hallucinations, and biases of GenAI, while enforcing public privacy and applicable state laws and policies. This bill requires state entities to collaborate with organizations that represent state government employees and industry experts in developing and providing training.
Finally, borrowing from the White House’s Blueprint, this bill requires any state agency or department that utilizes GenAI to directly communicate with a person, either through an online interface or telephonically, to clearly and in a conspicuous manner identify to that person that the person’s interaction with the state is being communicated through AI. Any state entity utilizes such a technology must provide on that entity’s internet website clear instructions informing the person how to directly communicate with a person from the state agency or department.
Suggested Amendments.
Amendment #1
Insert: (a)“Artificial Intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
Amendment #2
(B) The guidelines required by this paragraph shall build on guidance from the White House publication titled Blueprint for an Al Bill of Rights and the National Institute of Standards and Technology’s Al Risk Management Framework, or succeeding document, and shall address safety, algorithmic discrimination, data privacy, and notice of when materials are generated by generative artificial intelligence.
Amendment #3
Delete: 11549.64(i)(1) By July 1, ____, any state agency or department shall consider pilot projects of generative artificial intelligence applications, in consultation with organizations that represent state government employees and appropriate experts from academia and industry.
Prior/Related Legislation
SB 892 (Padilla, 2024) requires CDT to develop and adopt regulations to create an AI risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an ADS unless the contract includes a clause that provides a complete risk assessment, as specified. (Pending in the Senate Governmental Organization Committee)
SB 893 (Padilla, 2024) requires the establishment of the California Artificial Intelligence Research Hub in the Government Operations Agency to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development that seeks to harness the technology’s full potential for public benefit, as specified. (Pending in the Senate Government Organization Committee)
SB 942 (Becker, 2024) the California AI Transparency Act, among other things, requires a covered provider to create an AI detection tool by which a person can query the covered provide as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a GenAI system, as specified. (Pending in the Senate Judiciary Committee)
SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, this bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. (Pending in the Senate Governmental Organization Committee)
AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)
AB 3058 (Low, 2024) establishes the California Unconditional Benefit Income Pilot Program for the purpose of providing assistance to individuals who are unemployed because of automation or AI, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)
SB 313 (Dodd, 2023) would have established the Office of Artificial Intelligence in state government and require state agencies to disclose when they are using generative AI to communicate with a person and to provide them an option to speak with a natural person at the agency. (Held on the Senate Appropriations Committee Suspense File)
SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights,” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.
AB 302 (Ward, Chapter 800, Statutes of 2023) requires CDT, in coordination with other interagency bodies, to conduct a comprehensive inventory of all high-risk ADS used by state agencies on or before September 1, 2024, and report the findings to the Legislature by January 1, 2025, and annually thereafter, as specified.
FISCAL EFFECT: Appropriation: Fiscal Com.: Local:
SUPPORT:
Secure Justice
OPPOSITION:
None received
ARGUMENTS IN SUPPORT: In support of the bill, Secure Justice writes that, “[i]n this era of disinformation and distrust in our civic institutions, we believe such efforts like SB 896 will at minimum mitigate, if not outright eliminate, many of the known concerns arising from the use of AI.”
DUAL REFERRAL: Senate Governmental Organization Committee & Senate Judiciary Committee
Discussed in Hearing
Assembly Floor
Senate Standing Committee on Governmental Organization
Bill Author
Bill Co-Author(s):