Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

This bill would require the Department of Technology to develop and adopt regulations to create an automated decision system (ADS) procurement standard. To develop those regulations, the bill would require the department to consider principles and industry standards addressed in specified publications regarding AI risk management. The bill would require the ADS procurement standard to include, among other things, a detailed risk assessment procedure that analyzes specified characteristics of the ADS, methods for appropriate risk controls, as provided, and adverse incident monitoring procedures. The bill would require the department to, among other things, collaborate with specified organizations to develop the ADS procurement standard and review and update the ADS procurement standard and related regulations, as specified.

Commencing January 1, 2027, this bill would prohibit a state agency from procuring an ADS, entering into a contract for an ADS, or any service that utilizes an ADS, as specified, until the department has adopted regulations creating an ADS procurement standard. Commencing January 1, 2027, the bill would also require a contract for an ADS or a service that utilizes an ADS, as specified, to include a clause that, among other things, provides a completed risk assessment of the relevant ADS, as specified, requires adherence to appropriate risk controls, and provides procedures for adverse incident monitoring.

Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

Existing law, the California Consumer Privacy Act of 2018 (CCPA), grants to a consumer various rights with respect to personal information, as defined, that is collected by a business, as defined, including the right to request that a business delete personal information about the consumer that the business has collected from the consumer. Existing law, the California Privacy Rights Act of 2020, an initiative measure approved by the voters as Proposition 24 at the November 3, 2020, statewide general election, amended, added to, and reenacted the CCPA. The CCPA, as amended, establishes the California Privacy Protection Agency with full administrative power, authority, and jurisdiction to implement and enforce the CCPA.

This bill would require the Department of Technology to develop and adopt regulations to create an automated decision tool (ADT) system (ADS) procurement standard, following the adoption of specified regulations by the California Privacy Protection Agency or the enactment of comprehensive ADT legislation, whichever occurs sooner. standard. To develop those regulations, the bill would require the department to consider principles and industry standards addressed in specified publications regarding AI risk management. The bill would require the ADT ADS procurement standard to include, among other things, a detailed risk assessment procedure that analyzes specified characteristics of the ADT, ADS, methods for appropriate risk controls, as provided, and adverse incident monitoring procedures. The bill would require the department to, among other things, collaborate with specified organizations to develop the ADT ADS procurement standard and review and update the ADT ADS procurement standard and related regulations, as specified.

Commencing January 1, 2027, this bill would prohibit a state agency from procuring an ADT, ADS, entering into a contract for an ADT, ADS, or any service that utilizes an ADT ADS, as specified, until the department has adopted regulations creating an ADT ADS procurement standard. The Commencing January 1, 2027, the bill would also require a contract for an ADT ADS or a service that utilizes an ADT ADS, as specified, to include a clause that, among other things, provides a completed risk assessment of the relevant ADT, ADS, as specified, requires adherence to appropriate risk controls, and provides procedures for adverse incident monitoring.

Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

This bill would require the Department of Technology to develop and adopt regulations to create an artificial intelligence (AI) risk management standard, as specified. automated decision tool (ADT) procurement standard, following the adoption of specified regulations by the California Privacy Protection Agency or the enactment of comprehensive ADT legislation, whichever occurs sooner. To develop those regulations, the bill would authorize require the department to apply consider principles and industry standards addressed in specified publications regarding AI risk management. The bill would require the AI risk management ADT procurement standard to include, among other things, a detailed risk assessment procedure for procuring automated decision systems (ADS), as defined, that analyzes specified characteristics of the ADS, ADT, methods for appropriate risk controls, as provided, and adverse incident monitoring procedures. The bill would require the department to, among other things, collaborate with specified organizations to develop the AI risk management standard. ADT procurement standard and review and update the ADT procurement standard and related regulations, as specified.

This bill would, commencing six months after the date on which the regulations described in the paragraph above are approved and final, would prohibit a state agency from procuring an ADT, entering into a contract for an ADS, ADT, or any service that utilizes an ADS, unless the contract includes ADT until the department has adopted regulations creating an ADT procurement standard. The bill would also require a contract for an ADT or a service that utilizes an ADT to include a clause that, among other things, provides a completed risk assessment of the relevant ADS, ADT, as specified, requires adherence to appropriate risk controls, and provides procedures for adverse incident monitoring.

Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

This bill would require the Department of Technology to develop and adopt regulations to create an artificial intelligence (AI) risk management standard, consistent with specified publications regarding AI risk management, and in accordance with the rulemaking provisions of the Administrative Procedure Act. as specified. To develop those regulations, the bill would authorize the department to apply principles and industry standards addressed in specified publications regarding AI risk management. The bill would require the AI risk management standard to include, among other things, a detailed risk assessment procedure for procuring automated decision systems (ADS), as defined, that analyzes specified characteristics of the ADS, methods for appropriate risk controls, as provided, and adverse incident monitoring procedures. The bill would require the department to to, among other things, collaborate with specified organizations to develop the AI risk management standard.

This bill would, commencing six months after the date on which the regulations described in the paragraph above are approved and final, prohibit a state agency from entering into a contract for an ADS, or any service that utilizes an ADS, unless the contract includes a clause that, among other things, provides a completed risk assessment of the relevant ADS, as specified, requires adherence to appropriate risk controls, and provides procedures for adverse incident monitoring.

Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

This bill would require the Department of Technology to develop and adopt regulations to create an artificial intelligence (AI) risk management standard, consistent with specified publications regarding AI risk management, and in accordance with the rulemaking provisions of the Administrative Procedure Act. The bill would require the AI risk management standard to include, among other things, a detailed risk assessment procedure for procuring automated decision systems (ADS), as defined, that analyzes specified characteristics of the ADS, methods for appropriate risk controls, as provided, and adverse incident monitoring procedures. The bill would require the department to collaborate with specified organizations to develop the AI risk management standard.

This bill would, commencing on six months after the date on which the regulations described in the paragraph above are approved and final, prohibit a state agency from entering into a contract for an ADS, or any service that utilizes an ADS, unless the contract includes a clause that, among other things, provides a completed risk assessment of the relevant ADS, requires adherence to appropriate risk controls, and provides procedures for adverse incident monitoring.

Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

Existing law requires all contracts for the acquisition of information technology goods and services related to information technology projects, as defined, to be made by or under the supervision of the Department of Technology. Existing law requires all other contracts for the acquisition of information technology goods or services to be made by or under the supervision of the Department of General Services. Under existing law, both the Department of Technology and the Department of General Services are authorized to delegate their authority to another agency, as specified.

This bill would require the Department of Technology to establish safety, privacy, and nondiscrimination standards relating to artificial intelligence services, as defined. Commencing August 1, 2025, the bill would prohibit a contract for artificial intelligence services, as defined, from being entered into by the state unless the provider meets those standards.

This bill would require the Department of Technology to report to the Legislature regarding the standards it establishes, as specified.

 VETO  

Bill No: SB 892

Author: Padilla (D), et al.

Enrolled: 9/5/24

Vote: 27

SENATE GOVERNMENTAL ORG. COMMITTEE: 14-0, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Jones, Nguyen, Ochoa Bogh, Padilla, Portantino, Rubio, Seyarto, Smallwood-Cuevas

NO VOTE RECORDED: Ashby, Roth

SENATE JUDICIARY COMMITTEE: 11-0, 4/16/24

AYES: Umberg, Wilk, Allen, Ashby, Caballero, Durazo, Laird, Min, Niello, Stern, Wahab

SENATE APPROPRIATIONS COMMITTEE: 7-0, 5/16/24

AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab

SENATE FLOOR: 37-0, 5/23/24

AYES: Alvarado-Gil, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NO VOTE RECORDED: Allen, Archuleta, Newman

SENATE FLOOR: 40-0, 8/31/24

AYES: Allen, Alvarado-Gil, Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Newman, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

ASSEMBLY FLOOR: 75-0, 8/28/24 - See last page for vote

SUBJECT: Public contracts: automated decision systems: procurement standards

SOURCE: Author

DIGEST:

This bill requires the California Department of Technology (CDT) to develop and adopt regulations to create an automated decision system (ADS) procurement standard, as specified, and prohibits a state agency from procuring ADS, entering into a contract for ADS, or any service that utilizes ADS, until CDT has adopted regulations creating an ADS procurement standards, as specified.

ANALYSIS:

Existing law:

1. Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the CDT, as specified.
2. Governs, through the Administrative Procedures Act (APA), the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law.
3. Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk ADS that have been proposed for use or are being used, developed, or procured by state agencies, as specified.

This bill:

1. Requires CDT, to develop regulations related to the ADS procurement standard, and to consider principles and industry standards addressed in relevant publications, including, but not limited to, all of the following:
   1. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People (Blueprint), as specified.
   2. The Artificial Intelligence Risk Management Framework (AI RMF 1.0), as specified.
   3. The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), as specified.
   4. The Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence Memorandum, as specified.
2. Enumerates what the ADS procurement standard shall include, as specified; and, identifies requirements that CDT, in developing the ADS procurement standard, must do, as specified.
3. Requires CDT to adopt regulations pursuant to this bill, as specified, and requires that the regulations adopted by CDT pursuant to this bill do not contradict with regulations adopted by the California Privacy Protection Agency, and/or statewide legislation that establishes a regulatory framework governing the development and deployment of ADSs, as specified.
4. Requires CDT, commencing January 1, 2026, and annually thereafter, to update both of the following: the ADS procurement standard, and regulations adopted pursuant to this bill.
5. Prohibits a state agency, commencing January 1, 2027, from entering into a contract for an ADS, or entering into a contract for any service that utilizes an ADS, prior to the adoption of regulations by CDT, as specified.
6. Authorizes a state agency, commencing January 1, 2027, to enter into a contract for an ADS, or a service that utilizes ADS, only after CDT has adopted regulations pursuant to this bill and only if the contract includes a related clause that, among other things, provides a completed risk assessment of the relevant ADS, as specified.
7. Specifies that the above requirements to do not apply to projects approved before January 1, 2027, through the annual budget process.
8. Defines key terms related to the bill, as specified.

Background

Author Statement. According to the author’s office, “artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now.”

The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.

The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.

National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST was founded in 1901 and is now part of the U.S. Department of Commerce. NIST is one of the nation’s oldest physical science laboratories. Today, NIST measurements support the smallest of technologies to the largest and most complex of human-made creations – from nanoscale devices to tiny that tens of thousands can fit on the end of a single human hair up to earthquake-resistant skyscrapers and global communication networks.

Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). In 2024, the AI Procurement Lab (an independent, woman-owned, non-profit organization that advises government organizations) and the Center for Inclusive Change (a woman-owned small business that contributes AI related research and education programs) released their joint RMF PAIS. The two organizations worked together to create the framework which primarily focuses on risk management for high-risk systems. More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person’s safety, civil rights, and/or fundamental human rights and dignity.

California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to move forward with draft ADS technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students. According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”

Administrative Procedures Act and California’s Regulatory Processes. The Office of Administrative Law (OAL) is responsible for ensuring that California state agencies comply with the rulemaking procedures and standards set forth in California’s Administrative Procedures Act (APA). A “regulation” is any rule, regulation, order or standard of general application or the amendment, supplement, or revision of any rule, regulation, order, or standard adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it.

When adopting regulations, every department, division, office, officer, bureau, board or commission in the executive branch of the California state government must follow the rulemaking procedures in the APA and regulations adopted by the OAL, unless expressly exempted by statute from some or all of these requirements.

The APA requirements are designed to provide the public with a meaningful opportunity to participate in the adoption of regulations or rules that have the force of law by California state agencies and to ensure the creation of an adequate record for the OAL and judicial review.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Assembly Appropriations Committee, costs (General Fund) to CDT for additional staff and consultants. In the first year of implementation, CDT reports costs of $245,000 for one temporary program manager position, $2 million for external consultants, and $500,000 in GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $1.4 million for six permanent positions, $1.5 million for external consultants, and $300,000 in GenAI talent practices. CDT notes it is challenging to hire GenAI talent in the public sector because salaries are often two or three times higher in the private sector. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees.

Costs (General Fund) to the Department of General Services (DGS) for additional staffing. DGS anticipates costs of $510,000 annually ongoing for three positions to consult with CDT, implement procurement policy and procedures, conduct outreach and training, develop new procurement reporting functionality, and revise Fi$Cal to accommodate information gathering and reporting.

Unknown, potentially significant fiscal impacts if this bill’s moratorium on state agency procurement and contracting for ADSs is imposed. Under this bill, a moratorium on state agency contracting and procurement will be imposed beginning January 1, 2027, unless CDT’s regulations are enacted by that date. If the moratorium is imposed, the state may experience some cost savings to the extent ADS procurement and contracting (and associated costs) are paused, but may also experience significant costs if services are disrupted or agencies’ long-term plans cannot proceed without ADSs. If agencies must discontinue use of currently operational ADSs when their contracts expire during the moratorium, disruption of services and related costs may be much more significant.

Costs (General Fund, special funds) of an unknown but significant amount to state agencies once ADS procurement standards are implemented. DGS reports there may be an overall increase in contracting costs due to the new requirements and risk associated with the bill. According to DGS, the cost for a contractor to do business with the state in compliance with this bill may increase and contractors typically pass these costs on to the state. Additionally, DGS anticipates that adding more steps in the procurement process will increase workload to affected agencies throughout the state.

SUPPORT: (Verified 9/24/24)

AI Procurement Lab
American Federation of Musicians
American Federation of Musicians, Local 7
Center for Inclusive Change
City of Long Beach
City of San Jose
Consumer Reports
Electronic Frontier Foundation
Greenlining Institute
San Diego Regional Chamber of Commerce
Secure Justice
Surveillance Resistance Lab
Techequity Collaborative

OPPOSITION: (Verified 9/24/24)

CalChamber
Computer & Communications Industry Association

TechCA

TechNet

ARGUMENTS IN SUPPORT: In support of the bill, the AI Procurement Lab writes that, “AI technology, procured and deployed responsibly, has the potential to revolutionize the delivery of government services to Californians. However, this technology can also pose a significant risk to the safety, privacy, and civil rights of Californians. To protect and build trust, those procuring and using AI systems have a responsibility to add benefit and diligent risk analysis prior to procurement with exacting standards in risk management unique to the AI category. This is especially true when government officials sign contracts for administrative processes that perform tasks displacing their previously exercised discretion. An AI rsk management standard is also necessary when front-end government employees depend on the opaque logic of these systems that bears little or no resemblance to the reasoning processes of agency personnel.”

ARGUMENTS IN OPPOSITION: Opponents listed in opposition of this bill write jointly that, “[a]fter January 1, 2027, SB 892 bans the state from procuring ADS until regulations are completed by the California Department of Technology (CDT) and any contact for ADS must comply with the regulations… This bill is NOT based on a risk-based approach and will limit and increase the cost of many low-risk projects that could be used to deliver efficiencies both internally, as well as improve service delivery for constituents by leveraging technology that utilizes automated decision systems. This ban supposes that any procurement of ADT is concerning which is simply not true.”

GOVERNOR'S VETO MESSAGE:

This bill would task the California Department of Technology (CDT) to create an automated decision system procurement standard that complies with various requirements specified in the bill. Beginning on January 1, 2027, every state contract for an automated decision system would have to comply with these new regulations.

My administration shares the author’s commitment to ensuring the safe deployment of artificial intelligence (AI) and automated decision systems. Last year, I signed AB 302 (Ward, 2023), requiring CDT to conduct a comprehensive inventory of all high-risk automated decision systems that state agencies use, develop, or procure. And last September, I issued Executive Order 12-23, which, among other provisions, requires state agencies and departments to develop guardrails and deploy AI ethically and responsibly throughout state government to protect against potential risks.

Unfortunately, several aspects of this bill disrupt that ongoing work, including existing information technology modernization efforts, which would lead to implementation delays and higher expenses for critical projects. Additionally, this bill creates new and ongoing General Fund costs not included in the 2024 Budget Act.

In partnership with the Legislature this year, my Administration has enacted a balanced budget that avoids deep program cuts to vital services and protected investments in education, health care, climate, public safety, housing, and social service programs that millions of Californians rely on. It is important to remain disciplined when considering bills with significant fiscal implications that are not included in the budget, such as this measure.

ASSEMBLY FLOOR: 75-0, 8/28/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Megan Dahle, Davies, Dixon, Essayli, Flora, Mike Fong, Friedman, Gabriel, Gallagher, Garcia, Gipson, Grayson, Haney, Hart, Holden, Hoover, Irwin, Jackson, Jones-Sawyer, Kalra, Lackey, Lee, Low, Lowenthal, Maienschein, Mathis, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Pacheco, Papan, Joe Patterson, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Luz Rivas, Rodriguez, Blanca Rubio, Sanchez, Santiago, Schiavo, Soria, Ta, Ting, Valencia, Villapudua, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas

NO VOTE RECORDED: Bains, Cervantes, Ortega, Jim Patterson

Prepared by: Brian Duke / G.O. / (916) 651-1530

10/8/24 11:11:46

**** END ****

 UNFINISHED BUSINESS  

Bill No: SB 892

Author: Padilla (D), et al.

Amended: 8/19/24

Vote: 21

SENATE GOVERNMENTAL ORG. COMMITTEE: 14-0, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Jones, Nguyen, Ochoa Bogh, Padilla, Portantino, Rubio, Seyarto, Smallwood-Cuevas

NO VOTE RECORDED: Ashby, Roth

SENATE JUDICIARY COMMITTEE: 11-0, 4/16/24

AYES: Umberg, Wilk, Allen, Ashby, Caballero, Durazo, Laird, Min, Niello, Stern, Wahab

SENATE APPROPRIATIONS COMMITTEE: 7-0, 5/16/24

AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab

SENATE FLOOR: 37-0, 5/23/24

AYES: Alvarado-Gil, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NO VOTE RECORDED: Allen, Archuleta, Newman

ASSEMBLY FLOOR: 75-0, 8/28/24 - See last page for vote

SUBJECT: Public contracts: automated decision systems: procurement standards

SOURCE: Author

DIGEST:

This bill requires the California Department of Technology (CDT) to develop and adopt regulations to create an automated decision system (ADS) procurement standard, as specified, and prohibits a state agency from procuring ADS, entering into a contract for ADS, or any service that utilizes ADS, until CDT has adopted regulations creating an ADS procurement standards, as specified.

Assembly Amendments

among other things, require CDT to develop and adopt regulations to create an ADS procurement standard and, in developing that standard, to consider principles and industry standards addressed in relevant publications, as specified; requires CDT, in developing the ADS procurement standard to, among other things, consult with the California Privacy Protection Agency (CPPA); and, commencing January 1, 2027, prohibits a state agency from procuring an ADS, entering into a contract for an ADS, or entering into a contract for any service that utilizes an ADS, prior to the adoption of regulations by CDT, as specified.

ANALYSIS:

Existing law:

1. Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the CDT, as specified.
2. Governs, through the Administrative Procedures Act (APA), the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law.
3. Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk ADS that have been proposed for use or are being used, developed, or procured by state agencies, as specified.
4. The California Consumer Privacy Act of 2018 (CCPA) grants to a consumer various rights with respect to personal information, as defined, that is collected by a business, as defined, including the right to request that a business delete personal information about the consumer that the businesses has collected from the consumer, as specified.

This bill:

1. Defines “artificial intelligence” or “AI” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
2. Defines “automated decision system” or “ADS” to mean a computational process derived from machine learning, statistical modeling, data analytics, or AI that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons. “ADS” does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
3. Requires CDT, to develop regulations related to the ADS procurement standard, and to consider principles and industry standards addressed in relevant publications, including, but not limited to, all of the following:
   1. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People (Blueprint), published by the White House Office of Science and Technology Policy in October 2022.
   2. The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology (NIST) in January 2023.
   3. The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for the Inclusive Change in 2024.
   4. The Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence Memorandum, published by the Executive Office of the President, Office of Management and Budget, dated March 28, 2024.
4. Requires the ADS procurement standard to include all of the following:
   1. A detailed risk assessment procedure, as specified.
   2. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
   3. Adverse incident monitoring procedures.
   4. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
   5. A detailed equity assessment, as specified.
   6. An assessment that analyzes the level of human oversight associated with the use of ADS.
   7. Adherence to data minimization standards, including that an ADS vendor shall only use information provided by or obtained from an agency to provide the specific service authorized by the agency. Further, the data collected may not be used for training of proprietary vendor or third-party systems.
5. Requires CDT, in developing the ADS procurement standard, to do all of the following:
   1. Collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institutions focused on responsible ADS procurement, design, and deployment.
   2. Consult with the California Privacy Protection Agency (CPPA).
   3. Solicit public comment on the ADS procurement standard.
6. Requires CDT to adopt regulations pursuant to this bill, as specified, and requires that the regulations adopted by CDT pursuant to this bill do not contradict with either of the following:
   1. Regulations adopted by the CPPA, as specified.
   2. Statewide legislation that establishes a regulatory framework governing the development and deployment of ADSs.
7. Requires CDT, commencing January 1, 2026, and annually thereafter, to update both of the following: the ADS procurement standard, and regulations adopted pursuant to this bill.
8. Prohibits a state agency, commencing January 1, 2027, from entering into a contract for an ADS, or entering into a contract for any service that utilizes an ADS, prior to the adoption of regulations by CDT, as specified.
9. Authorizes a state agency, commencing January 1, 2027, to enter into a contract for an ADS, or a service that utilizes ADS, only after CDT has adopted regulations pursuant to this bill and only if the contract includes a related clause that, among other things, provides a completed risk assessment of the relevant ADS, as specified.
10. Specifies that the above requirements to do not apply to projects approved before January 1, 2027, through the annual budget process.

Background

Author Statement. According to the author’s office, “artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now.”

Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.

The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.

The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.

National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST was founded in 1901 and is now part of the U.S. Department of Commerce. NIST is one of the nation’s oldest physical science laboratories. Today, NIST measurements support the smallest of technologies to the largest and most complex of human-made creations – from nanoscale devices to tiny that tens of thousands can fit on the end of a single human hair up to earthquake-resistant skyscrapers and global communication networks.

Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). In 2024, the AI Procurement Lab (an independent, woman-owned, non-profit organization that advises government organizations) and the Center for Inclusive Change (a woman-owned small business that contributes AI related research and education programs) released their joint RMF PAIS. The two organizations worked together to create the framework which primarily focuses on risk management for high-risk systems. More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person’s safety, civil rights, and/or fundamental human rights and dignity.

California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to move forward with draft ADS technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students. According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”

Administrative Procedures Act and California’s Regulatory Processes. The Office of Administrative Law (OAL) is responsible for ensuring that California state agencies comply with the rulemaking procedures and standards set forth in California’s Administrative Procedures Act (APA). A “regulation” is any rule, regulation, order or standard of general application or the amendment, supplement, or revision of any rule, regulation, order, or standard adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it.

When adopting regulations, every department, division, office, officer, bureau, board or commission in the executive branch of the California state government must follow the rulemaking procedures in the APA and regulations adopted by the OAL, unless expressly exempted by statute from some or all of these requirements.

The APA requirements are designed to provide the public with a meaningful opportunity to participate in the adoption of regulations or rules that have the force of law by California state agencies and to ensure the creation of an adequate record for the OAL and judicial review.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Assembly Appropriations Committee, costs (General Fund) to CDT for additional staff and consultants. In the first year of implementation, CDT reports costs of $245,000 for one temporary program manager position, $2 million for external consultants, and $500,000 in GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $1.4 million for six permanent positions, $1.5 million for external consultants, and $300,000 in GenAI talent practices. CDT notes it is challenging to hire GenAI talent in the public sector because salaries are often two or three times higher in the private sector. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees.

Costs (General Fund) to the Department of General Services (DGS) for additional staffing. DGS anticipates costs of %510,000 annually ongoing for three positions to consult with CDT, implement procurement policy and procedures, conduct outreach and training, develop new procurement reporting functionality, and revise Fi$Cal to accommodate information gathering and reporting.

Unknown, potentially significant fiscal impacts if the bill’s moratorium on state agency procurement and contracting for ADSs is imposed. Under the bill, a moratorium on state agency contracting and procurement will be imposed beginning January 1, 2027, unless CDT’s regulations are enacted by that date. If the moratorium is imposed, the state may experience some cost savings to the extent ADS procurement and contracting (and associated costs) are paused, but may also experience significant costs if services are disrupted or agencies’ long-term plans cannot proceed without ADSs. If agencies must discontinue use of currently operational ADSs when their contracts expire during the moratorium, disruption of services and related costs may be much more significant.

Costs (General Fund, special funds) of an unknown but significant amount to state agencies once ADS procurement standards are implemented. DGS reports there may be an overall increase in contracting costs due to the new requirements and risk associated with the bill. According to DGS, the cost for a contractor to do business with the state in compliance with this bill may increase and contractors typically pass these costs on to the state. Additionally, DGS anticipates that adding more steps in the procurement process will increase workload to affected agencies throughout the state.

SUPPORT: (Verified 8/27/24)

AI Procurement Lab
American Federation of Musicians
American Federation of Musicians, Local 7
Center for Inclusive Change
City of Long Beach
City of San Jose
Consumer Reports
Electronic Frontier Foundation
Greenlining Institute
San Diego Regional Chamber of Commerce
Secure Justice
Surveillance Resistance Lab
Techequity Collaborative

OPPOSITION: (Verified 8/27/24)

CalChamber
Computer & Communications Industry Association

TechCA

TechNet

ARGUMENTS IN SUPPORT: In support of the bill, the AI Procurement Lab writes that, “AI technology, procured and deployed responsibly, has the potential to revolutionize the delivery of government services to Californians. However, this technology can also pose a significant risk to the safety, privacy, and civil rights of Californians. To protect and build trust, those procuring and using AI systems have a responsibility to add benefit and diligent risk analysis prior to procurement with exacting standards in risk management unique to the AI category. This is especially true when government officials sign contracts for administrative processes that perform tasks displacing their previously exercised discretion. An AI rsk management standard is also necessary when front-end government employees depend on the opaque logic of these systems that bears little or no resemblance to the reasoning processes of agency personnel.”

ASSEMBLY FLOOR: 75-0, 8/28/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Megan Dahle, Davies, Dixon, Essayli, Flora, Mike Fong, Friedman, Gabriel, Gallagher, Garcia, Gipson, Grayson, Haney, Hart, Holden, Hoover, Irwin, Jackson, Jones-Sawyer, Kalra, Lackey, Lee, Low, Lowenthal, Maienschein, Mathis, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Pacheco, Papan, Joe Patterson, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Luz Rivas, Rodriguez, Blanca Rubio, Sanchez, Santiago, Schiavo, Soria, Ta, Ting, Valencia, Villapudua, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas

NO VOTE RECORDED: Bains, Cervantes, Ortega, Jim Patterson

ARGUMENTS IN OPPOSITION: Opponents of the bill write jointly that, “[a]fter January 1, 2027, SB 892 bans the state from procuring ADS until regulations are completed by the California Department of Technology (CDT) and any contact for ADS must comply with the regulations… This bill is NOT based on a risk-based approach and will limit and increase the cost of many low-risk projects that could be used to deliver efficiencies both internally, as well as improve service delivery for constituents by leveraging technology that utilizes automated decision systems. This ban supposes that any procurement of ADT is concerning which is simply not true.”

Further, “SB 892 ties CDT regulations to the California Privacy and Protection Agency (CPPA) by stating that CDT regulations cannot be inconsistent with CPPA regulations and ‘similarly comprehensive statewide legislation that establishes a regulatory framework governing the development and deployment of ADTs.’ This requirement ties the hands of the state to make procurement decisions and gives power over state procurement to state legislation and CPPA regulations that have not been developed with state services in mind. CPPA is an independent consumer privacy organization focused on private businesses’ interaction with consumers. They are not an expert on state procurement and have no authority under Proposition 24 to regulate state procurement of technology or ADT. The Privacy Agency has not yet moved its risk assessment and ADMT regulations to formal rulemaking and even the Privacy Agency Board remains divided over the draft regulations.”

Finally, “SB 892 also tries, we believe unsuccessfully, to exempt current ongoing projects by not applying the procurement ban to projects that are approved through state budget processes prior to January 1, 2027. As with all state contracting, state departments and agencies have specific rules and requirements depending on project type and scope and not every project is individually ‘approved’ through the annual budget process. This means SB 892 could impact projects the state is already exploring about how AI can be used to protect vulnerable road users; reduce roadway congestion; improve health-care facility inspections; improve language access to state services; and enhance customer service.”

Click here to enter text. Prepared by: Brian Duke / G.O. / (916) 651-1530

8/29/24 10:23:39

**** END ****

 UNFINISHED BUSINESS  

Bill No: SB 892

Author: Padilla (D), et al.

Amended: 8/19/24

Vote: 21

SENATE GOVERNMENTAL ORG. COMMITTEE: 14-0, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Jones, Nguyen, Ochoa Bogh, Padilla, Portantino, Rubio, Seyarto, Smallwood-Cuevas

NO VOTE RECORDED: Ashby, Roth

SENATE JUDICIARY COMMITTEE: 11-0, 4/16/24

AYES: Umberg, Wilk, Allen, Ashby, Caballero, Durazo, Laird, Min, Niello, Stern, Wahab

SENATE APPROPRIATIONS COMMITTEE: 7-0, 5/16/24

AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab

SENATE FLOOR: 37-0, 5/23/24

AYES: Alvarado-Gil, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NO VOTE RECORDED: Allen, Archuleta, Newman

ASSEMBLY FLOOR: 58-0, 8/28/24 – Roll call vote not available

SUBJECT: Public contracts: automated decision systems: procurement standards

SOURCE: Author

DIGEST:

This bill requires the California Department of Technology (CDT) to develop and adopt regulations to create an automated decision system (ADS) procurement standard, as specified, and prohibits a state agency from procuring ADS, entering into a contract for ADS, or any service that utilizes ADS, until CDT has adopted regulations creating an ADS procurement standards, as specified.

Assembly Amendments

among other things, require CDT to develop and adopt regulations to create an ADS procurement standard and, in developing that standard, to consider principles and industry standards addressed in relevant publications, as specified; requires CDT, in developing the ADS procurement standard to, among other things, consult with the California Privacy Protection Agency (CPPA); and, commencing January 1, 2027, prohibits a state agency from procuring an ADS, entering into a contract for an ADS, or entering into a contract for any service that utilizes an ADS, prior to the adoption of regulations by CDT, as specified.

ANALYSIS:

Existing law:

1. Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the CDT, as specified.
2. Governs, through the Administrative Procedures Act (APA), the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law.
3. Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk ADS that have been proposed for use or are being used, developed, or procured by state agencies, as specified.
4. The California Consumer Privacy Act of 2018 (CCPA) grants to a consumer various rights with respect to personal information, as defined, that is collected by a business, as defined, including the right to request that a business delete personal information about the consumer that the businesses has collected from the consumer, as specified.

This bill:

1. Defines “artificial intelligence” or “AI” to mean an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
2. Defines “automated decision system” or “ADS” to mean a computational process derived from machine learning, statistical modeling, data analytics, or AI that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons. “ADS” does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
3. Requires CDT, to develop regulations related to the ADS procurement standard, and to consider principles and industry standards addressed in relevant publications, including, but not limited to, all of the following:
   1. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People (Blueprint), published by the White House Office of Science and Technology Policy in October 2022.
   2. The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology (NIST) in January 2023.
   3. The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for the Inclusive Change in 2024.
   4. The Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence Memorandum, published by the Executive Office of the President, Office of Management and Budget, dated March 28, 2024.
4. Requires the ADS procurement standard to include all of the following:
   1. A detailed risk assessment procedure, as specified.
   2. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
   3. Adverse incident monitoring procedures.
   4. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
   5. A detailed equity assessment, as specified.
   6. An assessment that analyzes the level of human oversight associated with the use of ADS.
   7. Adherence to data minimization standards, including that an ADS vendor shall only use information provided by or obtained from an agency to provide the specific service authorized by the agency. Further, the data collected may not be used for training of proprietary vendor or third-party systems.
5. Requires CDT, in developing the ADS procurement standard, to do all of the following:
   1. Collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institutions focused on responsible ADS procurement, design, and deployment.
   2. Consult with the California Privacy Protection Agency (CPPA).
   3. Solicit public comment on the ADS procurement standard.
6. Requires CDT to adopt regulations pursuant to this bill, as specified, and requires that the regulations adopted by CDT pursuant to this bill do not contradict with either of the following:
   1. Regulations adopted by the CPPA, as specified.
   2. Statewide legislation that establishes a regulatory framework governing the development and deployment of ADSs.
7. Requires CDT, commencing January 1, 2026, and annually thereafter, to update both of the following: the ADS procurement standard, and regulations adopted pursuant to this bill.
8. Prohibits a state agency, commencing January 1, 2027, from entering into a contract for an ADS, or entering into a contract for any service that utilizes an ADS, prior to the adoption of regulations by CDT, as specified.
9. Authorizes a state agency, commencing January 1, 2027, to enter into a contract for an ADS, or a service that utilizes ADS, only after CDT has adopted regulations pursuant to this bill and only if the contract includes a related clause that, among other things, provides a completed risk assessment of the relevant ADS, as specified.
10. Specifies that the above requirements to do not apply to projects approved before January 1, 2027, through the annual budget process.

Background

Author Statement. According to the author’s office, “artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now.”

Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.

The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.

The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.

National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST was founded in 1901 and is now part of the U.S. Department of Commerce. NIST is one of the nation’s oldest physical science laboratories. Today, NIST measurements support the smallest of technologies to the largest and most complex of human-made creations – from nanoscale devices to tiny that tens of thousands can fit on the end of a single human hair up to earthquake-resistant skyscrapers and global communication networks.

Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). In 2024, the AI Procurement Lab (an independent, woman-owned, non-profit organization that advises government organizations) and the Center for Inclusive Change (a woman-owned small business that contributes AI related research and education programs) released their joint RMF PAIS. The two organizations worked together to create the framework which primarily focuses on risk management for high-risk systems. More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person’s safety, civil rights, and/or fundamental human rights and dignity.

California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to move forward with draft ADS technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students. According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”

Administrative Procedures Act and California’s Regulatory Processes. The Office of Administrative Law (OAL) is responsible for ensuring that California state agencies comply with the rulemaking procedures and standards set forth in California’s Administrative Procedures Act (APA). A “regulation” is any rule, regulation, order or standard of general application or the amendment, supplement, or revision of any rule, regulation, order, or standard adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it.

When adopting regulations, every department, division, office, officer, bureau, board or commission in the executive branch of the California state government must follow the rulemaking procedures in the APA and regulations adopted by the OAL, unless expressly exempted by statute from some or all of these requirements.

The APA requirements are designed to provide the public with a meaningful opportunity to participate in the adoption of regulations or rules that have the force of law by California state agencies and to ensure the creation of an adequate record for the OAL and judicial review.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Assembly Appropriations Committee, costs (General Fund) to CDT for additional staff and consultants. In the first year of implementation, CDT reports costs of $245,000 for one temporary program manager position, $2 million for external consultants, and $500,000 in GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $1.4 million for six permanent positions, $1.5 million for external consultants, and $300,000 in GenAI talent practices. CDT notes it is challenging to hire GenAI talent in the public sector because salaries are often two or three times higher in the private sector. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees.

Costs (General Fund) to the Department of General Services (DGS) for additional staffing. DGS anticipates costs of %510,000 annually ongoing for three positions to consult with CDT, implement procurement policy and procedures, conduct outreach and training, develop new procurement reporting functionality, and revise Fi$Cal to accommodate information gathering and reporting.

Unknown, potentially significant fiscal impacts if the bill’s moratorium on state agency procurement and contracting for ADSs is imposed. Under the bill, a moratorium on state agency contracting and procurement will be imposed beginning January 1, 2027, unless CDT’s regulations are enacted by that date. If the moratorium is imposed, the state may experience some cost savings to the extent ADS procurement and contracting (and associated costs) are paused, but may also experience significant costs if services are disrupted or agencies’ long-term plans cannot proceed without ADSs. If agencies must discontinue use of currently operational ADSs when their contracts expire during the moratorium, disruption of services and related costs may be much more significant.

Costs (General Fund, special funds) of an unknown but significant amount to state agencies once ADS procurement standards are implemented. DGS reports there may be an overall increase in contracting costs due to the new requirements and risk associated with the bill. According to DGS, the cost for a contractor to do business with the state in compliance with this bill may increase and contractors typically pass these costs on to the state. Additionally, DGS anticipates that adding more steps in the procurement process will increase workload to affected agencies throughout the state.

SUPPORT: (Verified 8/27/24)

AI Procurement Lab
American Federation of Musicians
American Federation of Musicians, Local 7
Center for Inclusive Change
City of Long Beach
City of San Jose
Consumer Reports
Electronic Frontier Foundation
Greenlining Institute
San Diego Regional Chamber of Commerce
Secure Justice
Surveillance Resistance Lab
Techequity Collaborative

OPPOSITION: (Verified 8/27/24)

CalChamber
Computer & Communications Industry Association

TechCA

TechNet

ARGUMENTS IN SUPPORT: In support of the bill, the AI Procurement Lab writes that, “AI technology, procured and deployed responsibly, has the potential to revolutionize the delivery of government services to Californians. However, this technology can also pose a significant risk to the safety, privacy, and civil rights of Californians. To protect and build trust, those procuring and using AI systems have a responsibility to add benefit and diligent risk analysis prior to procurement with exacting standards in risk management unique to the AI category. This is especially true when government officials sign contracts for administrative processes that perform tasks displacing their previously exercised discretion. An AI rsk management standard is also necessary when front-end government employees depend on the opaque logic of these systems that bears little or no resemblance to the reasoning processes of agency personnel.”

ARGUMENTS IN OPPOSITION: Opponents of the bill write jointly that, “[a]fter January 1, 2027, SB 892 bans the state from procuring ADS until regulations are completed by the California Department of Technology (CDT) and any contact for ADS must comply with the regulations… This bill is NOT based on a risk-based approach and will limit and increase the cost of many low-risk projects that could be used to deliver efficiencies both internally, as well as improve service delivery for constituents by leveraging technology that utilizes automated decision systems. This ban supposes that any procurement of ADT is concerning which is simply not true.”

Further, “SB 892 ties CDT regulations to the California Privacy and Protection Agency (CPPA) by stating that CDT regulations cannot be inconsistent with CPPA regulations and ‘similarly comprehensive statewide legislation that establishes a regulatory framework governing the development and deployment of ADTs.’ This requirement ties the hands of the state to make procurement decisions and gives power over state procurement to state legislation and CPPA regulations that have not been developed with state services in mind. CPPA is an independent consumer privacy organization focused on private businesses’ interaction with consumers. They are not an expert on state procurement and have no authority under Proposition 24 to regulate state procurement of technology or ADT. The Privacy Agency has not yet moved its risk assessment and ADMT regulations to formal rulemaking and even the Privacy Agency Board remains divided over the draft regulations.”

Finally, “SB 892 also tries, we believe unsuccessfully, to exempt current ongoing projects by not applying the procurement ban to projects that are approved through state budget processes prior to January 1, 2027. As with all state contracting, state departments and agencies have specific rules and requirements depending on project type and scope and not every project is individually ‘approved’ through the annual budget process. This means SB 892 could impact projects the state is already exploring about how AI can be used to protect vulnerable road users; reduce roadway congestion; improve health-care facility inspections; improve language access to state services; and enhance customer service.”

Click here to enter text. Prepared by: Brian Duke / G.O. / (916) 651-1530

8/28/24 22:09:19

**** END ****

SENATE THIRD READING

STR Bill Id:SB 892

Author:(Padilla)

As Amended Ver:August 19, 2024

Majority vote

SUMMARY

Charges the Department of Technology (CDT) with crafting a procurement standard for the acquisition of automated decision systems (ADS), and imposes a moratorium on the procurement of new ADS until that standard is in place beginning in 2027.

Major Provisions

1. Defines ADS to mean computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. Excludes from this definition a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
2. Requires CDT to develop and adopt regulations to create an ADS procurement standard for the acquisition of ADS by the state. Requires that these regulations do not contradict regulations adopted by the California Privacy Protection Agency with respect to business' use of ADS, and any statewide legislation that establishes a regulatory framework governing the development and deployment of ADS.
3. Requires CDT to annually review and update the ADS procurement standard and any regulations adopted.
4. Beginning in 2027, prohibits a state agency from procuring an ADS, entering into a contract for an ADT, or entering into a contract for any service that utilizes an ADS, until CDT has adopted its regulations.
5. Beginning in 2027, after CDT has adopted regulations, prohibits a state agency from entering into a contract for an ADS or entering into a contract for a service that uses an ADS unless the contract includes a risk assessment, a requirement to adhere to procurement standards, and procedures for adverse incident monitoring, among other required contract provisions.

COMMENTS

Background. Over the past thirty years, ADS have become ubiquitous in society. ADS are enabled by artificial intelligence, and when they are embedded into a decisionmaking process they can increase efficiency and productivity many-fold compared to an equivalent human worker. But ADS share many faults with the humans they replace. If they are not thoroughly tested, the decisions they produce can be incorrect or even biased. This becomes especially problematic when an ADS is used to make a consequential decision: for example, to determine whether to hire an employee, or prioritize a patient seeking healthcare, or evaluate a student's scholarship application.

What this bill would do. This bill would require CDT to develop a set of procurement standards for ADS, and would impose a moratorium on the procurement of ADS until these standards are in place.

Analysis. California's state government regularly uses ADS to engage in consequential decisionmaking. The premises of this bill are, first, that the State should be much more informed about the ADS it procures, and second, that vendors of ADS ought to be much more conscious and deliberate about what the impacts of their tools might be. In short, California ought to get a handle on ADS while it still can before they become yet another phenomenon—like social media, greenhouse gases, guns, and urban sprawl—that could have been dealt with deliberately and intelligently in the past, but was instead allowed to expand without any constraints past the point of being manageable.

According to the Author

The public has been left vulnerable to the dangers [Artificial Intelligence] AI poses because of congressional failure to act and the regulatory desert they've created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead…But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values. By doing so, we can use California's purchasing power to influence the development of appropriate standards around automated-decision making systems.

Arguments in Support

Electronic Frontier Foundation writes:

SB 892 rightly recognizes the potential risks of AI technology alongside its potential for fostering innovation. It sets the foundation for a framework that reflects principles that are necessary to create an environment that the public can trust, including requirements to:

1. Assess the risks associated with procuring high risk AI systems and ADS, particularly in areas that have a high impact on individual rights, including but not limited to health, education, employment, insurance, utilities, critical infrastructure, public services, and justice/legal.
2. Require the development and adoption of risk management standard that includes risk assessment, appropriate risk controls, and adverse incident monitoring when procuring such systems.
3. Require engagement and consultation with a diversity of stakeholders, including those most affected by government use of these systems, such as public trust and safety experts, community-based organizations, civil society groups, workers, academic researchers.
4. Ensure people have access to robust notices and opportunities to appeal or opt out of high-risk AI system outcomes or ADS decisions.

The City of San Jose writes:

The City of San José has established itself as a leader in AI, having convened the nationwide GovAI Coalition, which is comprised of more than 600 officials at 250 local, county, and state governments to promote the responsible use of AI. While the City has been an early supporter and thought partner on the benefits of AI, we also recognize the potential risks and the need for legislation and regulation to provide guardrails throughout its development. This legislation strikes the appropriate balance of assessing and minimizing risks while not hindering growth. AI has the potential to provide incredible societal benefits if harnessed appropriately, and California has an immense opportunity to lead this effort in partnership with technology companies and Silicon Valley leaders.

Secure Justice writes:

As the fourth largest economy in the world, California must ensure that innovation does not come at the expense of our constituents. Given the state's purchasing power through our procurement of technology and services, California can drive the innovation of AI technology and services by requiring that any technology and services contracted by the state adhere to standards that protect Californians. While the Governor's Executive Order to Prepare California for the Progress of Artificial Intelligence required the issuance of general guidelines for public sector procurement, California must take further steps to safeguard the critical services we provide.

Arguments in Opposition

None.

FISCAL COMMENTS

1. Costs (General Fund) to CDT for additional staff and consultants. In the first year of implementation, CDT reports costs of $254,000 for one temporary program manager position, $2 million for external consultants, and $500,000 in GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $1.4 million for six permanent positions, $1.5 million for external consultants, and $300,000 in GenAI talent practices. CDT notes it is challenging to hire GenAI talent in the public sector because salaries are often two or three times higher in the private sector. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees.
2. Costs (General Fund) to the Department of General Services (DGS) for additional staffing. DGS anticipates costs of $510,000 annually ongoing for three positions to consult with CDT, implement procurement policy and procedures, conduct outreach and training, develop new procurement reporting functionality, and revise Fi$Cal to accommodate information gathering and reporting.
3. Unknown, potentially significant fiscal impacts if the bill's moratorium on state agency procurement and contracting for ADSs is imposed. Under the bill, a moratorium on state agency contracting and procurement will be imposed beginning January 1, 2027, unless CDT's regulations are enacted by that date. If the moratorium is imposed, the state may experience some cost savings to the extent ADS procurement and contracting (and associated costs) are paused, but may also experience significant costs if services are disrupted or agencies' long-term plans cannot proceed without ADSs. If agencies must discontinue use of currently operational ADSs when their contracts expire during the moratorium, disruption of services and related costs may be much more significant.
4. Costs (General Fund, special funds) of an unknown but significant amount to state agencies once ADS procurement standards are implemented. DGS reports there may be an overall increase in contracting costs due to the new requirements and risk associated with the bill. According to DGS, the cost for a contractor to do business with the state in compliance with this bill may increase and contractors typically pass these costs on to the state. Additionally, DGS anticipates that adding more steps in the procurement process will increase workload to affected agencies throughout the state.

VOTES

SENATE FLOOR: 37-0-3

YES: Alvarado-Gil, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dahle, Dodd, Durazo, Eggman, Glazer, Gonzalez, Grove, Hurtado, Jones, Laird, Limón, McGuire, Menjivar, Min, Nguyen, Niello, Ochoa Bogh, Padilla, Portantino, Roth, Rubio, Seyarto, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

ABS, ABST OR NV: Allen, Archuleta, Newman

ASM PRIVACY AND CONSUMER PROTECTION: 8-0-3

YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson

ABS, ABST OR NV: Joe Patterson, Dixon, Hoover

ASM APPROPRIATIONS: 12-0-3

YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart, Pellerin, Ta, Villapudua

ABS, ABST OR NV: Sanchez, Dixon, Jim Patterson

UPDATED

VERSION: August 19, 2024

CONSULTANT: Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0004150

Date of Hearing: August 7, 2024

ASSEMBLY COMMITTEE ON APPROPRIATIONS

Buffy Wicks, Chair

SBPCA Bill Id:SB 892

Author:(Padilla) – As Amended Ver:July 3, 2024

Urgency: No State Mandated Local Program: No Reimbursable: No

SUMMARY:

This bill requires the California Department of Technology (CDT) to issue regulations that set a standard for state procurement of automated decision-making tools (ADTs), and prohibits a state agency from contracting for or procuring an ADT until CDT’s regulations are enacted.

The bill defines ADT to mean an artificial intelligence (AI) system or service that makes a consequential decision, or is a substantial factor in making consequential decisions. The bill defines consequential decision to include decisions related to certain aspects of employment, education and vocational training, essential utilities, adoption, health care, financial services, and legal services, among other specified subjects.

Among other provisions, the bill:

1. Requires CDT to develop and adopt regulations to create an ADT procurement standard following the adoption of ADT regulations by the California Privacy Protection Agency (Privacy Agency), or the passage of similarly comprehensive legislation by the Legislature.
2. Requires the ADT procurement standard developed by CDT to include all of the following:
3. A detailed risk assessment procedure.
4. Methods for appropriate risk controls between a state agency and ADT vendor.
5. Adverse incident monitoring procedures.
6. Identification and classification of prohibited use cases and applications of ADT the state may not procure.
7. A detailed equity assessment.
8. An assessment analyzing the level of human oversight associated with use of the ADT.
9. Adherence to data minimization standards.
10. Requires CDT’s regulations be consistent with specified ADT regulations adopted by the Privacy Agency, as well as any similarly comprehensive legislation passed by the Legislature.
11. Requires CDT to annually review and update the ADT procurement standard and any regulations adopted.
12. Prohibits a state agency from procuring an ADT, entering into a contract for an ADT, or entering into a contract for any service that utilizes an ADT, until CDT has adopted its regulations.
13. After CDT has adopted regulations, prohibits a state agency from entering into a contract for an ADT or entering into a contract for a service that uses an ADT unless the contract includes a risk assessment, a requirement to adhere to procurement standards, and procedures for adverse incident monitoring, among other required contract provisions.

FISCAL EFFECT:

1. Costs (General Fund) to CDT for additional staff and consultants. In the first year of implementation, CDT reports costs of $254,000 for one temporary program manager position, $2 million for external consultants, and $500,000 in GenAI talent practices to train state employees. In subsequent years, CDT anticipates costs of $1.4 million for six permanent positions, $1.5 million for external consultants, and $300,000 in GenAI talent practices. CDT notes it is challenging to hire GenAI talent in the public sector because salaries are often two or three times higher in the private sector. As a result, CDT reports, its fiscal estimates for this bill include significant funding for external consultants, and actual costs for consultants may be higher if the state does not provide sufficient funding for training of state employees.
2. Minor and absorbable costs to the Privacy Agency to consult on CDT’s regulations.
3. Costs (General Fund) to the Department of General Services (DGS) for additional staffing. DGS anticipates costs of $510,000 annually ongoing for three positions to consult with CDT, implement procurement policy and procedures, conduct outreach and training, develop new procurement reporting functionality, and revise Fi$Cal to accommodate information gathering and reporting.
4. Unknown but likely significant fiscal impacts due to the bill’s moratorium on state agency procurement and contracting for ADTs. It is unknown when CDT’s regulations will be adopted, ending the moratorium. In the meantime, the state may experience some cost savings to the extent ADT procurement and contracting (and associated costs) are paused, but may also experience significant costs if services are disrupted or agencies’ long-term plans cannot proceed without ADTs. If agencies must discontinue use of currently operational ADTs when their contracts expire during the prohibition, disruption of services and related costs may be much more significant.
5. Costs (General Fund, special funds) of an unknown but significant amount to state agencies once ADT procurement standards are implemented. DGS reports there may be an overall increase in contracting costs due to the new requirements and risk associated with the bill. According to DGS, the cost for a contractor to do business with the state in compliance with this bill may increase and contractors typically pass these costs on to the state. Additionally, DGS anticipates that adding more steps in the procurement process will increase workload to affected agencies throughout the state.

According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years. 

COMMENTS:

1. Purpose. According to the author:

The public has been left vulnerable to the dangers AI poses because of congressional failure to act and the regulatory desert they’ve created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead…But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values. By doing so, we can use California’s purchasing power to influence the development of appropriate standards around automated-decision making systems.

1. Background. ADTs are increasingly used to help companies and government agencies make decisions. In 2023, the White House released its Blueprint for an AI Bill of Rights, stating:

Too often, these tools are used to limit our opportunities and prevent our access to critical resources or services. These problems are well documented. In America and around the world, systems supposed to help with patient care have proven unsafe, ineffective, or biased. Algorithms used in hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed new harmful bias and discrimination.

This bill requires CDT to develop a standard for state agency procurement of specified ADTs and issue related regulations, and imposes a prohibition on state agency procurement of or contracting for ADTs until CDT’s regulations are adopted. The bill specifies that CDT’s procurement regulations shall be developed and adopted following adoption of ADT regulations by the Privacy Agency. The Privacy Agency is in the process of developing regulations to facilitate exercise of a consumer’s right to access and opt out of a business’s use of ADT. Draft regulations were discussed at the Privacy Agency’s July 2024 board meeting but the board did not advance the draft regulations.

Alternatively, under this bill, CDT may develop and adopt its procurement regulations following enactment of “comprehensive statewide legislation” establishing a regulatory framework for ADTs. The analysis of this bill by the Assembly Committee on Privacy and Consumer Protection points to AB 2930 (Bauer-Kahan), of this legislative session, as an example of such legislation. AB 2930 requires entities that deploy ADTs (including state and local government agencies) to perform algorithmic discrimination risk assessments regarding ADTs before using them to make consequential decisions. AB 2930 also requires entities to provide notice to the public about use of ADTs, the ability to opt out of use of ADTs, and an ability to appeal a consequential decision made by an ADT. AB 2930 is currently pending in the Senate Appropriations Committee.

Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081

Date of Hearing: July 2, 2024

ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION

Rebecca Bauer-Kahan, Chair

SBPCA Bill Id:SB 892

Author:(Padilla) – As Amended Ver:June 21, 2024

AS PROPOSED TO BE AMENDED

SENATE VOTE: 37-0

SUBJECT: Public contracts: automated decision systems: AI risk management standards

SYNOPSIS

Over the past thirty years, automated decision tools (ADTs) have become ubiquitous in society. ADTs are enabled by artificial intelligence, and when they are embedded into a decisionmaking process they can increase efficiency and productivity many-fold compared to an equivalent human worker. But ADTs share many faults with the humans they replace. If they are not thoroughly tested, the decisions they produce can be incorrect or even biased. This becomes especially problematic when an ADT is used to make a consequential decision: for example, to determine whether to hire an employee, or prioritize a patient seeking healthcare, or evaluate a student’s scholarship application.

California’s state government regularly uses ADTs to engage in consequential decisionmaking. This bill, as proposed to be amended, would require the Department of Technology (CDT) to implement a procurement standard for ADTs in order to assess these tools for bias before they can affect the lives of Californians. Committee amendments would make this bill consistent with Proposition 24 (2020), which granted the California Privacy Protection Agency (Privacy Agency) regulatory authority over businesses’ use of automated decisionmaking technologies. A second Committee amendment would align various definitions in this bill with those in a related bill, AB 2930 (Bauer-Kahan, 2024.) Finally, a Committee amendment would prohibit the procurement of ADTs by the state until CDT has developed and implemented necessary procurement standards. The amendments are set forth in full below.

This bill is author-sponsored and supported by a variety of cities and advocacy groups, including Oakland Privacy, Techequity Collaborative, and the Electronic Frontier Foundation. It has no opposition. If this bill passes out of this committee it will next be heard by the Assembly Appropriations Committee.

SUMMARY: Charges CDT with crafting a procurement standard for the acquisition of ADTs, and imposes a moratorium on the procurement of new ADTs until that standard is in place. Specifically, this bill:

1. Defines “artificial intelligence” to mean “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.”
2. Defines “automated decision tool” (ADT) to mean “an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.”
3. Defines “consequential decision” to mean a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, any of the following:
4. Employment with respect to all of the following:
5. Pay or promotion.
6. Hiring or termination.
7. Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
8. Education and vocational training as it relates to all of the following:
9. Assessment or placement.
10. Detecting student cheating or plagiarism.
11. Accreditation.
12. Certification.
13. Admissions or enrollment.
14. Discipline.
15. Evaluation.
16. Financial aid or scholarships.
17. Housing or lodging, including rental or short-term housing or lodging.
18. All of the following essential utilities:
19. Electricity.
20. Heat.
21. Water.
22. Internet or telecommunications access.
23. Transportation.
24. Family planning.
25. Adoption services, reproductive services, or assessments related to child protective services.
26. Health care or health insurance, including mental health care, dental, or vision.
27. Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
28. All of the following aspects of the criminal justice system:
29. Risk assessments for pretrial hearings.
30. Sentencing.
31. Parole.
32. Legal services.
33. Private arbitration.
34. Mediation.
35. Voting.
36. Defines “substantial factor” to mean an element of a decisionmaking process that is capable of altering the outcome of the process.
37. Requires CDT to develop and adopt regulations to create an ADT procurement standard following the adoption of regulations by the Privacy Agency related to business’ use of ADTs, or the passage of similarly comprehensive legislation by the Legislature. Requires CDT to consider principles and industry standards addressed in various relevant publications, as specified.
38. Requires the ADT procurement standard to include all of the following:
39. A detailed risk assessment procedure that analyzes:
40. Organizational and supply chain governance associated with the ADT.
41. The purpose and use of the ADT.
42. Any known potential misuses or abuses of the ADT.
43. An assessment of the legality, traceability, and provenance of the data the ADT uses and the legality of the output of the ADT.
44. The robustness, accuracy, and reliability of the ADT.
45. The interpretability and explainability of the ADT.
46. Methods for appropriate risk controls between the state agency and ADT vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
47. Adverse incident monitoring procedures.
48. Identification and classification of prohibited use cases and applications of ADT that the state shall not procure.
49. A detailed equity assessment, as specified.
50. An assessment that analyzes the level of human oversight associated with the use of the ADT.
51. Adherence to data minimization standards.
52. Requires CDT to do all of the following in developing the ADT procurement standard:
53. Collaborate with organizations that represent state and local government employees and industry experts, as specified.
54. Consult with the California Privacy Protection Agency.
55. Solicit public comment on the ADT procurement standard.
56. Requires any regulations adopted by CDT pursuant to this section be consistent with regulations adopted by the Privacy Agency pursuant to paragraph (16) of subdivision (a) of Section 1798.185 of the Civil Code, as well as any similarly comprehensive legislation passed by the Legislature.
57. Requires CDT to annually review and update the ADT procurement standard and any regulations adopted pursuant to this section.
58. Prohibits a state agency from procuring an ADT, entering into a contract for an ADT, or entering into a contract for any service that utilizes an ADT, until the department has adopted regulations pursuant to this section.
59. Prohibits a state agency from entering into a contract for an ADT or entering into a contract for any service that utilizes an ADT after CDT has adopted regulations pursuant to this section unless the contract includes a clause that does all of the following:
60. Provides a completed risk assessment, as specified.
61. Requires the state agency and/or ADT vendor, as appropriate, to adhere to appropriate procurement standards.
62. Provides procedures for adverse incident monitoring.
63. Requires authorization from the state agency before deployment of ADT upgrades and enhancements.
64. Requires the state agency or the ADT vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADT, and information about how to appeal or opt-out of ADT decisions or outcomes.
65. Provides a termination right in the event of a significant breach of responsibility or violation by the vendor.

EXISTING LAW:

1. Establishes the California Privacy Protection Agency and vests it with full administrative power, authority, and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018. (Civ. Code § 1798.199.10.)
2. Requires the Attorney General to issue regulations governing access and opt-out rights with respect to businesses’ use of automated decisionmaking technology. Passes along this authority to the California Privacy Protection Agency after a set period of time. (Civ. Code § 1798.185.)
3. Establishes CDT within the Government Operations Agency (GovOps), under the supervision of the Director of Technology, also known as the State Chief Information Officer. (Gov. Code § 11545(a).)
4. Expresses the intent of the Legislature that policies and procedures developed by CDT and Department of General Services (DGS) pertaining to the acquisition of IT goods and services provide for all of the following: the expeditious and value-effective acquisition of IT goods and services to satisfy state requirements; the acquisition of IT goods and services within a competitive framework; the delegation of authority by DGS to each state agency that has demonstrated to DGS’s satisfaction the ability to conduct value-effective IT goods and services acquisitions; and the review and resolution of protests submitted by any bidders with respect to any IT goods and services acquisitions. (Pub. Con. Code § 12101.)
5. Requires CDT, on or before September 1, 2024, to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. (Gov. Code § 11546.45.5(b).)
6. Defines the following terms:
7. “Automated decision system” (ADS) means a computational process derived from machine learning, statistical modeling, data analytics, or AI that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. ADS does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
8. “State agency” includes every state office, department, division, bureau, the California State University, the Board of Parole Hearings, and specified boards. It does not include the University of California, the Legislature, the judicial branch, or any board, except as provided. (Gov’t Code § 11546.45.5(a).)

FISCAL EFFECT: As currently in print, this bill is keyed fiscal.

COMMENTS:

1. Background. The term “automated decisionmaking tool,” or “ADT,” refers to computer systems that replace or supplement human decisionmaking ability. ADTs are used to decide whom to hire, what treatments patients should receive, whether to grant parole, and many other life-altering decisions that used to be made entirely by human beings.

Using an ADT is, in many cases, a good thing. Computers can generally draw upon far more digitally-encoded data, and process this data far more quickly, than any human being. A recent New York Times article described the development of a breast cancer screening system that uses artificial intelligence:

From the millions of cases the system is fed, the technology creates a mathematical representation of normal mammograms and those with cancers. With the ability to look at each image in a more granular way than the human eye, it then compares that baseline to find abnormalities in each mammogram.

Last year, after a test on more than 275,000 breast cancer cases, Kheiron [Medical Technologies, the system’s developer] reported that its A.I. software matched the performance of human radiologists when acting as the second reader of mammography scans. It also cut down on radiologists’ workloads by at least 30 percent because it reduced the number of X-rays they needed to read. In other results from a Hungarian clinic last year, the technology increased the cancer detection rate by 13 percent because more malignancies were identified.^[1]

However, ADTs are far from flawless. Care should be taken to avoid what New York University Professor Meredith Broussard describes as “the notion that computers are more ‘objective’ or ‘unbiased’ because they distill questions and answers down to mathematical evaluation…an unwavering faith that if the world used more computers, and used them properly, social problems would disappear and we’d create a digitally enabled utopia.”^[2] Many aspects of the real world are not reducible to data, and many complex situations present competing demands that human beings remain best equipped to assess.

ADT usage continues to increase. Global consulting firm McKinsey & Company reports that the adoption of artificial intelligence among the organizations it surveyed more than doubled between 2017 and 2022, with 50 percent now reporting they use the technology.^[3]

ADTs use artificial intelligence. ADTs use artificial intelligence (AI) to make consequential decisions. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. This is because AI is not fundamentally different from other computer functions. Its novelty lies in its application: unlike standard computer functions, AI is able to accomplish tasks that are normally performed by humans.

Bias and discrimination can result from poor training data. There is a famous saying in computer science: “garbage in, garbage out.” The performance of an ADT is directly impacted by the quality, quantity, and relevance of the data used to train it.^[4] If the data used to train the ADT is biased, the tool’s outputs will be similarly biased. Over the past thirty years, several industries have been forced to contend with this fact as they have attempted to introduce ADTs into their workflows. Specific examples follow:

Hiring and recruitment tools. It is no secret that people of various races, genders, and cultures are not distributed equally throughout the workforce. An ADT that is trained on historical data to make hiring decisions will be predisposed to maintain the ratios it is trained on; as described by Aditya Malik, the Founder and CEO of Valuematrix.ai:

Generative AI, for all its grandeur, has the potential to perpetuate latent biases inherited from human creators. A disconcerting echo of historical prejudices may inadvertently seep into the algorithms. Imagine a scenario where previous senior managers, driven by biases of gender, age, faith or race, rejected candidates for misguided reasons. The AI, if not vigilantly curated, might misconstrue these patterns as indicators of incompetence, thus exacerbating the exclusion of qualified candidates from underrepresented backgrounds.^[5]

This was notoriously experienced by Amazon, who considered automating their hiring practices in the early 2010s. They opted against this approach in 2015 when they realized that their ADT-enabled system was not rating candidates in a gender-neutral way. In fact, their system was excluding women from the pool of acceptable candidates because it had been trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of inequities across the tech industry.^[6]

Sentencing and bail decisions. ADTs are frequently used to inform sentencing and bail decisions. These tools are trained using historical data, and the predictions they make can therefore reflect historical bias. A 2016 Propublica study dove into the use of one such tool – COMPAS – in Broward County, Florida.^[7] The study determined that Black defendants were far more likely than white defendants to be incorrectly judged to be at a higher risk of recidivism, while white defendants were more likely than Black defendants to be incorrectly flagged as low risk.

The for-profit company that developed this tool, Northpointe, does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. These discrepancies mirror historical injustices perpetuated against Black Americans by California’s criminal justice system.^[8]

Healthcare. When ADTs are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. In 2007, an ADT was developed to help doctors estimate whether it was safe for people who had delivered previous children through cesarean section to deliver subsequent children vaginally – a risky procedure. The ADT considered various health relevant factors as it made its decision, such as the woman’s age, her reason for the previous cesarean, and how long ago the cesarean had been performed. However, a 2017 study found that the ADT was biased; it predicted Black and Latino people were less likely to have a successful vaginal birth after a cesarean than non-Hispanic white women. As a result, doctors performed more cesareans on Black and Latino people than on white people.^[9] These discrepancies perpetuate historical biases – Black Americans, for example, have historically received a lower standard of healthcare than their white counterparts.^[10]

Credit and loan approval. Financial tools that utilize ADTs are similarly susceptible to bias and discrimination. An investigation by The Markup (and co-published by the Associated Press) revealed that in 2019, lenders were more likely to deny home loans to people of color; in particular, lenders were 40 percent more likely to turn down Latino applicants for loans, 50 percent more likely to deny Asian/Pacific Islander applicants, and 70 percent more likely to deny Native American applicants than similar white applicants. Lenders were 80 percent more likely to reject Black applicants than similar white applicants. In every case, the prospective borrowers of color looked almost exactly the same on paper as the white applicants, except for their race.^[11]

1. What this bill would do. This bill would require CDT to develop a set of procurement standard for ADTs, and would impose a moratorium on the procurement of ADTs until these standards are in place. The bill requires CDT to draw from a variety of sources when creating the standards, including the California Privacy Protection Agency.

The premises of this bill are, first, that the State should be much more informed about the ADTs it procures, and second, that vendors of ADTs ought to be much more conscious and deliberate about what the impacts of their tools might be. In short, California ought to get a handle on ADTs while it still can before they become yet another phenomenon—like social media, greenhouse gases, guns, and urban sprawl—that could have been dealt with deliberately and intelligently in the past, but was instead allowed to expand without any constraints past the point of being manageable.

1. Author’s statement:

Artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now. The public has been left vulnerable to the dangers AI poses because of congressional failure to act and the regulatory desert they’ve created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead. We are the home of Silicon Valley, we are leading the way in AI development and innovation and we are also one of the largest purchasing entities of technology in the world. But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values. By doing so, we can use California’s purchasing power to influence the development of appropriate standards around automated-decision making systems.

1. Committee amendments. Committee amendments strengthen this bill by aligning it with other major ADT legislation and regulations, and by introducing a moratorium on the acquisition of new ADTs until safety measures have been enacted.

AB 2930 (Bauer-Kahan, 2024) would require developers and deployers of ADTs to perform risk assessments on their tools before using them to make consequential decisions. That bill defines a “state government deployer” to mean “a state government agency that uses an automated decision tool to make a consequential decision” – these are the same entities referred to in the current bill by the phrase “state agency.” Furthermore, “developers” in AB 2930 are the likely source of any ADTs procured under the terms of this bill. A Committee amendment has been introduced to align the definitions of “ADT,” “consequential decision,” and “substantial factor” between these two bills. The author has committed to maintaining this alignment as the two bills progress through the legislative process.

California’s Proposition 24 (2020) granted the California Privacy Protection Agency regulatory authority over access and opt-out rights with respect to businesses’ use of automated decisionmaking technologies. In order to avoid conflicting with this mandate, an amendment has been introduced to delay the implementation of procurement standards by the Department of Technology until after the Privacy Agency has adopted regulations pursuant to Section 1798.185 of the Civil Code, or until the legislature has adopted a similarly comprehensive framework (which is necessarily consistent with Prop 24.) Another amendment requires that any regulations adopted by CDT pursuant to the terms of this bill be consistent with regulations adopted by the Privacy Agency pursuant to Section 1798.185 of the Civil Code.

Finally, a Committee amendment has been introduced which takes seriously this bill’s premise: that ADTs are capable of great harm to individuals and society if they are not carefully assessed prior to deployment. According to the author:

The procurement of technology without proper guardrails has already led to adverse impacts to communities in various states and California. Recently, the Electronic Privacy Information Center (EPIC) released a report titled “Outsourced and Automated: How AI Companies Have Taken Over Government Decision-Making”, showcasing how many agencies across all fifty states have outsourced their services and decisions to private vendors. The technology includes anything from automated facial recognition systems, applicant screening systems, and fraud detention systems.

• The Michigan Integrated Data System (MiDAS) was launched in October 2013 as a replacement for a 30 year old system that the Michigan Unemployment Insurance Agency used to administer unemployment benefits. Simultaneously with the system’s rollout, the UI agency reduced its staff by one third. The system nearly immediately has significant issues and by the time the system was shut down in September 2015 under community and legal pressure, MiDAS had incorrectly accused more than 34,000 people of unemployment fraud, cutting critical receiving benefits, due to a poorly designed algorithm within a poorly designed system. Michigan state legislators have promised to seek at least $30 million in compensation for those falsely accused. Advocates are pushing for close to $100 million given the trauma faced by people who saw their credits and reputations ruined, filed for bankruptcy, had their houses foreclosed, or were made homeless.
• Use of the Allegheny Family Screening Tool in Pennsylvania which was meant to guide responses to calls to family service agency about potential child neglect. This is a predictive model that the ACLU analyzed for how its design functions as policy and its impact on families.
• In 2022, there was a Carnegie Mellon study conducted on the tool and a resulting AP investigation; included in that “CMU researchers found that from August 2016 to May 2018, the tool calculated scores that suggested 32.5% of Black children reported as being neglected should be subject to a “mandatory” investigation, compared with 20.8% of white children.” As a result of that investigation, it was reported last January that the DOJ is investigating the use of this algorithm.
• Another example is COMPAS which is a risk assessment model used to predict recidivism, which was found to be racially biased.
• In 2022, an LAO report found that California’s Employment Development Department (EDD), procured an automated review software that flagged 1.1 million claims as fraudulent, meaning worker’s stopped receive much needed benefit. Ultimately, more than half of those claims (600,000) were confirmed as legitimate.

As such, a Committee amendment has been introduced to prohibit the procurement of ADTs by the state until the procurement standard required by this bill has been developed and implemented. This moratorium does not prohibit the continued use of existing ADTs – only the procurement of new ADTs.

1. Full text of bill as proposed to be amended:

SECTION 1. Section 12100.1 is added to the Public Contract Code, to read:

12100.1. (a) For purposes of this section, the following definitions apply:

(1) “Artificial intelligence” or “AI” means an engineered or machine-based system that, varies in its level of autonomy and that can, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.

(2) (A) “Automated decision system” or “ADS” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.

(B) “Automated decision system” does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.

(2) “Automated decision tool” or “ADT” means an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.

(3) “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, any of the following:

(A) Employment with respect to all of the following:

(i) Pay or promotion.

(ii) Hiring or termination.

(iii) Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.

(B) Education and vocational training as it relates to all of the following:

(i) Assessment or placement.

(ii) Detecting student cheating or plagiarism.

(iii) Accreditation.

(iv) Certification.

(v) Admissions or enrollment.

(vi) Discipline.

(vii) Evaluation.

(viii) Financial aid or scholarships.

(C) Housing or lodging, including rental or short-term housing or lodging.

(D) All of the following essential utilities:

(i) Electricity.

(ii) Heat.

(iii) Water.

(iv) Internet or telecommunications access.

(v) Transportation.

(E) Family planning.

(F) Adoption services, reproductive services, or assessments related to child protective services.

(G) Health care or health insurance, including mental health care, dental, or vision.

(H) Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.

(I) All of the following aspects of the criminal justice system:

(i) Risk assessments for pretrial hearings.

(ii) Sentencing.

(iii) Parole.

(J) Legal services.

(K) Private arbitration.

(L) Mediation.

(M) Voting.

(3) “Department” means the Department of Technology.

(4) “Privacy Agency” means the California Privacy Protection Agency.

(4) “High-risk automated decision system” or “high-risk ADS” means an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, free speech, housing or accommodations, education, employment, credit, health care, child welfare, immigration, and criminal justice.

(5) “Substantial factor” means an element of a decisionmaking process that is capable of altering the outcome of the process.

(b) Following the adoption of regulations by the Privacy Agency pursuant to paragraph (16) of subdivision (a) of Section 1798.185 of the Civil Code, or following the enactment of similarly comprehensive statewide legislation that establishes a regulatory framework governing the development and deployment of ADTs, whichever occurs sooner, Tthe department shall develop and adopt regulations to create an AI risk management ADT procurement standard.

(1) To develop regulations related to the AI risk management ADT procurement standard, the department may apply shall consider principles and industry standards addressed in relevant publications, including, but not limited to, any all of the following:

(A) The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, published by the White House Office of Science and Technology Policy in October 2022.

(B) The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology (NIST) in January 2023.

(C) The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for Inclusive Change in 2024.

(D) The Advancing Governance, Innovation, and Risk Management for Agency Use of Artificial Intelligence Memorandum, published by the Executive Office of the President, Office of Management and Budget, dated March 28, 2024.

(2) The AI risk managementADT procurement standard shall include all of the following:

(A) A detailed risk assessment procedure for procuring ADS that analyzes all of the following:

(i) Organizational and supply chain governance associated with the ADSADT.

(ii) The purpose and use of the ADSADT.

(iii) Any known potential misuses or abuses of the ADSADT.

(iv) An assessment of the legality, traceability, and provenance of the data the ADSADT uses and the legality of the output of the ADSADT.

(v) The robustness, accuracy, and reliability of the ADSADT.

(vi) The interpretability and explainability of the ADSADT.

(B) Methods for appropriate risk controls between the state agency and ADSADT vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.

(C) Adverse incident monitoring procedures.

(D) Identification and classification of prohibited use cases and applications of ADSADT that the state shall not procure.

(E) A detailed equity assessment that analyzes, at a minimum, all of the following:

(i) The individuals and communities that will interact with the high-risk ADSADT.

(ii) How the information or decisions generated by the ADSADT will impact an individual’s rights, freedoms, economic status, health, health care, or well-being.

(iii) Any issues that may arise if the ADSADT is inaccurate.

(iv) How users of with diverse abilities will interact with the user interface of the ADSADT and whether the ADSADT integrates and interacts with commonly used assistive technologies.

(F) An assessment that analyzes the level of human oversight associated with the use of ADSADT.

(G) Adherence to data minimization standards, including that an AI or ADSADT vendor shall only use information provided by or obtained from an agency to provide the specific service authorized by the agency. Further, the data collected may not be used for training of proprietary vendor or third-party systems.

(3) To In developing the AI risk managementADT procurement standard, the department shall comply withdo all of the following:

(A) Collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institutions focused on responsible AI ADT procurement, design, and deployment.

(B) Consult with the California Privacy Protection Agency.

(C) Solicit public comment on the risk managementADT procurement standard.

(4) (A) Subject to subparagraph (B), The the department shall adopt regulations pursuant to this subdivision in accordance with the provisions of Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code.

(B) Any regulations adopted by the department pursuant to subparagraph (A) shall be consistent with both of the following:

(i) Any regulations adopted by the Privacy Agency pursuant to paragraph (16) of subdivision (a) of Section 1798.185 of the Civil Code.

(ii) Any statewide legislation that establishes a regulatory framework governing the development and deployment of ADTs.

(5) The department shall annually review and update both of the following:

(A) The ADT procurement standard.

(B) Any regulations adopted pursuant to this subdivision.

(c) A state agency shall not procure an ADT, enter into a contract for an ADT, or enter into a contract for any service that utilizes an ADT, until the department has adopted regulations pursuant to subdivision (b).

(dc) Commencing six months after the date on which the regulations described in subdivision (b) are approved and final, Aa state agency shall notmay enter into a contract for an automated decision systemADT, or any a service that utilizes an automated decision systemADT, unless after the department has adopted regulations pursuant to subdivision (b) only if the contract includes a clause that does all of the following:

(1) Provides a completed risk assessment of the relevant ADSADT that analyzes the items included in subparagraph (A) of paragraph (2) of subdivision (b).

(2) Requires the state agency or the ADSADT vendor, or both, to adhere to appropriate risk controlsprocurement standards.

(3) Provides procedures for adverse incident monitoring.

(4) Requires authorization from the state agency before deployment of ADSADT upgrades and enhancements.

(5) Requires the state agency or the ADSADT vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADSADT, and information about how to appeal or opt-out of ADSADT decisions or outcomes.

(6) Provides a termination right in the event of a significant breach of responsibility or violation by the vendor.

1. Related legislation. AB 2930 (Bauer-Kahan, 2024) would require developers and deployers of automated decision tools to perform impact assessments before using the tools, as well as disclose the tools use to affected consumers. This bill is currently pending in Senate Judiciary Committee.

AB 302 (Ward, Chap. 800, Stats. 2023) required CDT to conduct an inventory of all high-risk automated decision systems being used in state agencies. The bill was chaptered on October 8th, 2023.

SB 313 (Dodd, 2023) would have established an Office of Artificial Intelligence within CDT, with “the powers and authorities necessary to guide the design, use, or deployment of automated systems by a state agency to ensure that all AI systems are designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and that minimizes bias and promotes equitable outcomes for all Californians.” The bill was held in Senate Appropriations.

AB 2269 (Chau, 2020) would have established the Automated Decision Systems (ADS) Accountability Act of 2020. The Act would have required businesses in California that provide a person, as defined, with a program or device that uses an automated decision system (ADS) to take affirmative steps to ensure that there are processes in place to continually test for biases during the development and usage of the ADS, conduct an ADS impact assessment on its program or device to determine whether the ADS has a disproportionate adverse impact on a protected class, as specified, examine if the ADS in question serves reasonable objectives and furthers a legitimate interest, and compare the ADS to alternatives or reasonable modifications that may be taken to limit adverse consequences on protected classes. The bill was held in Assembly Privacy and Consumer Protection.

ARGUMENTS IN SUPPORT:

Electronic Frontier Foundation writes:

SB 892 rightly recognizes the potential risks of AI technology alongside its potential for fostering innovation. It sets the foundation for a framework that reflects principles that are necessary to create an environment that the public can trust, including requirements to:

• Assess the risks associated with procuring high risk AI systems and ADS, particularly in areas that have a high impact on individual rights, including but not limited to health, education, employment, insurance, utilities, critical infrastructure, public services, and justice/legal.
• Require the development and adoption of risk management standard that includes risk assessment, appropriate risk controls, and adverse incident monitoring when procuring such systems.
• Require engagement and consultation with a diversity of stakeholders, including those most affected by government use of these systems, such as public trust and safety experts, community-based organizations, civil society groups, workers, academic researchers.
• Ensure people have access to robust notices and opportunities to appeal or opt out of high-risk AI system outcomes or ADS decisions.

The City of San Jose writes:

The City of San José has established itself as a leader in AI, having convened the nationwide GovAI Coalition, which is comprised of more than 600 officials at 250 local, county, and state governments to promote the responsible use of AI. While the City has been an early supporter and thought partner on the benefits of AI, we also recognize the potential risks and the need for legislation and regulation to provide guardrails throughout its development. This legislation strikes the appropriate balance of assessing and minimizing risks while not hindering growth. AI has the potential to provide incredible societal benefits if harnessed appropriately, and California has an immense opportunity to lead this effort in partnership with technology companies and Silicon Valley leaders.

Secure Justice writes:

As the fourth largest economy in the world, California must ensure that innovation does not come at the expense of our constituents. Given the state’s purchasing power through our procurement of technology and services, California can drive the innovation of AI technology and services by requiring that any technology and services contracted by the state adhere to standards that protect Californians. While the Governor’s Executive Order to Prepare California for the Progress of Artificial Intelligence required the issuance of general guidelines for public sector procurement, California must take further steps to safeguard the critical services we provide.

REGISTERED SUPPORT / OPPOSITION:

Support

AI Procurement Lab
American Federation of Musicians, Local 7
City of Long Beach
City of San Jose
Electronic Frontier Foundation
Greenlining Institute; the
Oakland Privacy
San Diego Regional Chamber of Commerce
Secure Justice
TechEquity Collaborative

Opposition

None on file.

Analysis Prepared by: Slater Sharp / P. & C.P. / (916) 319-2200

1. Santariano & Metz, “Using A.I. to Detect Breast Cancer That Doctors Miss,” New York Times, Mar. 5, 2023, https://www.nytimes.com/2023/03/05/technology/artificial-intelligence-breast-cancer-detection.html. ↑

2. Broussard, “Artificial Unintelligence,” MIT Press, 2019. ↑

3. McKinsey & Co., “The state of AI in 2022—and a half decade in review,” Dec. 6, 2022, https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022-and-a-half-decade-in-review. ↑

4. Rohit Sehgal, “AI Needs Data More Than Data Needs AI”, Forbes, Oct. 5, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/10/05/ai-needs-data-more-than-data-needs-ai/. ↑

5. Aditya Malik, “AI Bias In Recruitment: Ethical Implications And Transparency,” Forbes, Sep. 25, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/09/25/ai-bias-in-recruitment-ethical-implications-and-transparency/. ↑

6. Jeffrey Dastin, “Amazon scraps secret AI recruiting tool that showed bias against women,” Reuters, Oct. 9, 2018, https://www.reuters.com/article/amazoncom-jobs-automation/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSL2N1VB1FQ/. ↑

7. Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, “Machine Bias”, Propublica, May 23, 2016, https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. ↑

8. California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” California. ↑

9. Caleb J Colón-Rodríguez, “Shedding Light on Healthcare Algorithmic and Artificial Intelligence Bias,” U.S. Department of Health & Human Services Office of Minority Health, Jul. 12, 2023, minorityhealth.hhs.gov/news/shedding-light-healthcare-algorithmic-and-artificial-intelligence-bias. ↑

10. California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” California. ↑

11. Emmanuel Martinez and Lauren Kirchner, “The Secret Bias Hidden in Mortgage-Approval Algorithms,” Markup, Aug. 25, 2021, themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms. ↑

 THIRD READING  

Bill No: SB 892

Author: Padilla (D)

Amended: 4/10/24

Vote: 21

SENATE GOVERNMENTAL ORG. COMMITTEE: 14-0, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Jones, Nguyen, Ochoa Bogh, Padilla, Portantino, Rubio, Seyarto, Smallwood-Cuevas

NO VOTE RECORDED: Ashby, Roth

SENATE JUDICIARY COMMITTEE: 11-0, 4/16/24

AYES: Umberg, Wilk, Allen, Ashby, Caballero, Durazo, Laird, Min, Niello, Stern, Wahab

SENATE APPROPRIATIONS COMMITTEE: 7-0, 5/16/24

AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab

SUBJECT: Public contracts: automated decision systems: AI risk management standards

SOURCE: Author

DIGEST:

This bill requires the California Department of Technology (CDT) to develop and adopt regulations to create an artificial intelligence (AI) risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, six months after the regulations are adopted, from contracting for an automated decision system (ADS) unless the contract includes a clause that provides a complete risk assessment, as specified.

ANALYSIS:

Existing law:

1. Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the CDT, as specified.
2. Governs, through the Administrative Procedures Act (APA), the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law.
3. Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk ADS that have been proposed for use or are being used, developed, or procured by state agencies, as specified.

This bill:

1. Requires CDT to develop and adopt regulations to create an AI risk management standard and requires the standard to be consistent with all of the following publications:
   1. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, published by the White House Office of Science and Technology Policy in October 2022.
   2. The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology in January 2023.
   3. The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for Inclusive Change in 2024.
2. Requires the AI risk management standard to include all of the following:
   1. A detailed risk assessment procedure for procuring ADS, as specified.
   2. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
   3. Adverse incident monitoring procedures.
   4. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
   5. An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.
3. Requires the CDT, in order to develop the AI risk management standard, collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researches, and research institutions focused on responsible AI procurement, design, and deployment.
4. Requires the CDT to adopt regulations pursuant to this bill in accordance with the provisions of the APA, as specified.
5. Prohibits, commencing six months after the date on which the regulations described above are approved and final, a state agency from entering into a contract for an ADS, or any service that utilizes an ADS, unless the contract includes a clause that does all of the following:
   1. Provides a completed risk assessment of the relevant ADS.
   2. Requires the state agency or the ADS vendor, or both, to adhere to appropriate risk controls.
   3. Provides procedures for adverse incident monitoring.
   4. Requires authorization from the state agency before deployment of ADS upgrades and enhancements.
   5. Requires the state agency or the ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, and information about how to appeal or opt-out of ADS decisions or outcomes.
6. Defines “artificial intelligence” to mean an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
7. Includes other related definitions.

Background

1. Author Statement. According to the author’s office, “artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now.”

Further, “the public has been left vulnerable to the dangers AI poses because of congressional failure to act and the regulatory desert they’ve created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead. We are the home of Silicon Valley, we are leading the way in AI development and innovation and we are also one of the largest purchasing entities of technology in the world. But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values.”

1. Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.
2. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint), identifying five principles to guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint serves as a guide to safeguard individuals from AI-related threats.

The five principles are: “Safe and Effective Systems,” which emphasizes consultation with diverse communities and rigorous testing to ensure safety and effectiveness; “Algorithmic Discrimination Protections,” which requires proactive measures to prevent unjustified disparate treatment based on protected characteristics; “Data Privacy,” which mandates design choices that protect privacy by default and respect user consent; “Notice and Explanation,” which calls for clear, accessible documentation and timely updates about the system’s functioning and outcomes; and “Human Alternatives, Consideration, and Fallback,” which ensures that users can opt out of automated systems in favor of human alternatives when appropriate. These principles aim to create a framework that promotes safety, equity, and transparency in the use of AI.

1. National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST, founded in 1901 and now part of the U.S. Department of Commerce, is one of the nation’s oldest physical science laboratories. Today, NIST supports a range of technologies from nanoscale devices to earthquake-resistant skyscrapers and global communication networks.

In January 2023, as directed by the National AI Initiative Act of 2020, NIST released the AI Risk Management Framework (AI RMF 1.0) to better manage AI-related risks to individuals, organizations, and society. This framework, intended for voluntary use, aims to improve the trustworthiness of AI products, services, and systems through a consensus-driven, open, and collaborative process. It aligns with other AI risk management efforts, highlighting the serious and unique risks posed by AI.

1. Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). In 2024, the AI Procurement Lab (an independent, woman-owned, non-profit organization that advises government organizations) and the Center for Inclusive Change (a woman-owned small business that contributes AI related research and education programs) released their joint RMF PAIS. The two organizations worked together to create the framework which primarily focuses on risk management for high-risk systems. More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person’s safety, civil rights, and/or fundamental human rights and dignity.
2. California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to advance draft regulations on ADS and risk assessment, focusing on how businesses use AI and collect personal information from consumers, workers, and students. The proposed rules aim to create guidelines for areas where AI and personal data influence Californians' lives, including job compensation, demotion, housing, insurance, health care, and student expulsion. For instance, job candidates could opt out of AI-based emotional state predictions during interviews without fear of discrimination.

The proposed rules require businesses to notify individuals before using AI. If people opt out of interacting with an AI model, businesses cannot discriminate against them. If people agree to use an AI service, businesses would be required to respond to requests about how personal information is used to make predictions. Employers and third-party contractors must also conduct risk assessments to evaluate their technology's performance. The CPPA will conduct a statewide informational outreach tour before initiating formal rulemaking later this year.

1. High-Risk Automated Decision Systems. AB 302 (Ward, Chapter 800, Statutes of 2023) directed CDT, in coordination with other interagency bodies as it deems appropriate, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. Per the bill, that inventory must be completed on or before September 1, 2024.
2. Administrative Procedures Act and California’s Regulatory Process. The Office of Administrative Law (OAL) ensures that California state agencies comply with the rulemaking procedures and standards set by California’s Administrative Procedures Act (APA). A "regulation" refers to any rule, regulation, order, or standard of general application, including amendments or revisions, adopted by a state agency to implement or interpret the law it enforces.

When adopting regulations, all executive branch entities of California state government must follow the APA rulemaking procedures and OAL regulations unless specifically exempted by statute. The APA is designed to provide the public with a meaningful opportunity to participate in the regulation adoption process and to ensure the creation of a comprehensive record for OAL and judicial review.

1. AI Risk Management Standard. This bill requires the CDT to develop and adopt regulations to create an AI risk management standard. Specifically, this bill requires the standard developed by CDT to be consistent with three identified publications: the White House’s Blueprint; NIST’s AI RMF 1.0; and the AI Procurement Lab and the Center for Inclusive Change co-produced RMF PAIS 1.0 – each discussed above.

To develop the standard, this bill requires CDT to collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institution focused on responsible AI procurement, design, and deployment. This bill specifies that CDT must adopt the regulations required by this measure in accordance with the provisions of the APA.

Six months after the regulations have been approved and final, this bill prohibits any state agency from entering into a contract for an ADS system, or any service that utilizes an ADS, unless the contract includes a clause that does all of the following:

• 1. Provides a completed risk assessment of the relevant ADS.
  2. Requires the state agency or the ADS vendor, or both, to adhere to appropriate risk controls.
  3. Provides procedures for adverse incident monitoring.
  4. Requires authorization from the state agency before deployment of ADS upgrades and enhancements.
  5. Requires the state agency or the ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, and information about how to appeal or opt-out of ADS decisions or outcomes.

Prior/Related Legislation

SB 896 (Dodd, 2024) the AI Accountability Act, among other things, requires specified state entities to produce a State of California Benefits and Risk of Generative AI Report, as specified; requires specified state entities to perform a joint risk analysis of potential threats posed by the use of generative AI to California’s critical energy infrastructure. (Pending on the Senate Floor)

SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, this bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. (Pending on the Senate Floor)

AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer, as defined, and a developer of an automated decision tool, to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. (Pending in the Assembly Appropriations Committee)

SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights,” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.

AB 302 (Ward, Chapter 800, Statutes of 2023) required CDT, in coordination with other interagency bodies, to conduct a comprehensive inventory of all high-risk ADS used by state agencies on or before September 1, 2024, and report the findings to the Legislature by January 1, 2025, and annually thereafter, as specified.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Senate Appropriations Committee, unknown significant costs ranging into the millions of dollars for CDT to develop the AI risk management standard. One-time costs may include workload to develop regulations and solicit feedback from state and local government employees and industry experts, community-based organizations, civil society groups, academic researchers, and research institutions. Ongoing costs may include providing training or other guidance for state agency procurement staff in applying the standard to contracts. To the extent that this work cannot be absorbed through CDT’s existing resources, there will be a General Fund impact.

Unknown fiscal impact to state procurement costs. To the extent that contractors incur increased costs to comply with the standard, then there will be an increase in state procurement costs as contractors generally pass increased business expenses to the state through increased contracting costs.

SUPPORT: (Verified 5/16/24)

AI Procurement Lab
American Federation of Musicians
Center for Inclusive Change
City of Long Beach
Consumer Reports
Electronic Frontier Foundation
Greenlining Institute
Secure Justice
Surveillance Resistance Lab
TechEquity Collaborative

OPPOSITION: (Verified 5/16/24)

None received

ARGUMENTS IN SUPPORT: In support of the bill, TechEquity writes, “California must ensure that innovation does not come at the expense of Californians. Given the state’s purchasing power through our procurement of technology and services, California has the opportunity to set a national standard that drives the innovation of AI technology and services by requiring that any technology and services contracted by the state adhere to standards that protect Californians. While the Governor’s Executive Order to Prepare California for the Progress of Artificial Intelligence took some early steps to set guidelines for public sector procurement, California must take further steps to safeguard the critical services we provide.”

Further, the AI Procurement Lab writes that, “AI, procured and deployed responsibly, has the potential to revolutionize the delivery of government services to Californians. However, this technology can also pose a significant risk to the safety, privacy, and civil rights of Californians. To protect and build trust, those procuring and using AI systems have a responsibility to add benefit and diligent risk analysis prior to procurement with exacting standards in risk management unique to the AI category. This is especially true when government officials sign contracts for administrative processes that perform tasks displacing their previously exercised discretion. An AI risk management standard is also necessary when front-end government employees depend on the opaque logic of these systems that bears little or no resemblance to the reasoning process of agency personnel.”

Prepared by: Brian Duke / G.O. / (916) 651-1530

5/17/24 10:09:08

**** END ****

SENATE COMMITTEE ON

Senator Anna Caballero, Chair

2023 - 2024 Regular Session

SB 892 (Padilla) - Public contracts: automated decision systems: AI risk management standards

Bill Summary:

SB 892 requires the California Department of Technology (CDT) to develop and adopt regulations for an artificial intelligence (AI) risk management standard regarding state procurement and use of automated decision systems (ADS), as specified.

Fiscal Impact:

• Unknown, significant costs ranging into the millions of dollars for CDT to develop the AI risk management standard. One-time costs may include workload to develop regulations and solicit feedback from state and local government employees and industry experts, community-based organizations, civil society groups, academic researchers, and research institutions. Ongoing costs may include providing training or other guidance for state agency procurement staff in applying the standard to contracts. To the extent that this work cannot be absorbed through CDT’s existing resources, there will be a General Fund impact.
• Unknown fiscal impact to state procurement costs. To the extent that contractors incur increased costs to comply with the standard, then there will be an increase in state procurement costs as contractors generally pass increased business expenses to the state through increased contracting costs.

Background:

Generally, AI is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises. Some of these most recent efforts by government and other business entities include:

• The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People (Blueprint). Released in October of 2022 by the White House’s Office of Science and Technology Policy, the Blueprint identifies five principles that should guide the design, use, and deployment of ADSs and is intended to be a guide to protect all people from AI-related threats.
• National Institute of Standards and Technology’s (NIST) AI Risk Management Framework (AI RMF 1.0). Released in January of 2023, the AI RMF 1.0 is intended for voluntary use by organizations and individuals (known as AI actors) for the responsible design, development, deployment, and use of AI systems over time. The AI RMF 1.0 is divided into two parts. Part 1 discusses how organizations can frame the risks related to AI, how AI risk and trustworthiness are analyzed, and the characteristics of trustworthy AI systems. Part 2 describes four specific functions to help organizations address the risks of AI systems in practice.
• Executive Order (EO) N-12-23. Governor Gavin Newsom issued EO N-12-23 in September of 2023 “to study the development, use, and risks of AI technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government.”
• Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). The RMF PAIS 1.0 was authored by the AI Procurement Lab and the Center for Inclusive Change in 2024. The RMF PAIS 1.0 aims to “provide organizations and procurement teams with an essential tool that classifies the risks embedded within each procurement opportunity for the purposes of risk awareness, assessment, measurement, mitigation, treatment, control, monitoring, and management.”

Proposed Law:

• Requires CDT to develop and adopt regulations to create an AI risk management standard.
• Requires the standard to be consistent with the Blueprint; the AI RMF 1.0, and the RMF PAIS 1.0, authored by the AI Procurement Lab and the Center for Inclusive Change in 2024.
• Requires the standard to include:
  • A detailed risk assessment procedure for procuring ADS that analyzes organizational and supply chain governance associated with the ADS, the purpose and use of the ADS, any known potential misuses or abuses of the ADS, an assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS; the robustness, accuracy, and reliability of the ADS; and the interpretability and explainability of the ADS.
  • Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
  • Adverse incident monitoring procedures.
  • Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
  • An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.
• Requires CDT to collaborate with organizations that represent state and local government employees and industry experts including public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institutions focused on responsible AI procurement, design, and deployment.
• Six months after CDT’s regulations for the AI risk management standard are approved and final, prohibits a state agency from entering into a contract for an ADS or any service that utilizes an ADS unless the contract includes a clause that provides all of the following:
  • A completed risk assessment of the relevant ADS; a requirement for the state agency and vendor to adhere to appropriate risk controls; procedures for adverse incident monitoring; authorization from the state agency before deployment of ADS upgrades and enhancements; and a requirement for providing notification to individuals that would likely be affected by the decisions or outcomes of the ADS and information on opting out of ADS decisions or outcomes.
• Establishes definitions.

Related Legislation:

SB 893 (Padilla, 2024) establishes the California Artificial Intelligence Research Hub in the Government Operations Agency to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development, as specified. SB 893 is pending in this committee.

SB 896 (Dodd, 2024) establishes the Artificial Intelligence Accountability Act, which tasks specified state entities with developing and periodically evaluating general guidelines for the use, regulation, and identification of potential risks of incorporating AI and generative AI (GenAI) within state government operations. SB 896 is pending in this committee.

SB 942 (Becker, 2024) the California AI Transparency Act, among other things, requires a covered provider to create an AI detection tool by which a person can query the covered provide as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a GenAI system, as specified. SB 942 is pending in this committee.

SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, SB 1047 establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. SB 1047 is pending in this committee.

-- END --

SENATE JUDICIARY COMMITTEE

Senator Thomas Umberg, Chair

2023-2024 Regular Session

Bill No: SB 892 (Author:Padilla)

Version: April 10, 2024

Hearing Date: April 16, 2024

Fiscal: Yes

Urgency: No

Consultant: CK

SUBJECT

Public contracts: automated decision systems: AI risk management standards

DIGEST

This bill requires the Department of Technology (CDT) to establish an artificial intelligence (AI) risk management standard regarding procurement and use of automated decionmaking systems (ADS) that is informed by leading established standards. The standard must detail specified procedures for assessing and controlling risks, prohibited use cases, and an assessment for impact on vulnerable communities. State agencies are prohibited from entering into contracts for ADS unless the contract contains specified provisions.

EXECUTIVE SUMMARY

The transformative power of AI and specifically generative AI (GenAI) is unquestionable, and it offers numerous benefits for society, including state government. It can enhance efficiency and effectiveness in various sectors freeing up human resources. Specifically, the deployment of ADS, algorithm-driven applications that assist or replace human discretionary decisionmaking, can improve decision-making processes by analyzing vast amounts of data to identify patterns, trends, and potential insights. However, as with most technologies, there are also inherent risks and challenges. One major concern is the potential for bias in ADS applications, especially when high-risk ADS are determining individuals access to housing, credit, employment, or other benefits. Ensuring transparency, accountability, and fairness in ADS is crucial to mitigate these risks and maintain public trust.

This bill directs CDT to establish an AI risk management standard, modeled after mainstream, widely-accepted publications, to guide the procurement, use, and oversight of ADS in government agencies. Agencies are required to include specific clauses in contracts for the procurement of ADS after CDT promulgates attendant regulations. This bill is author-sponsored. It is supported by a number of civil rights and advocacy organizations. No timely opposition has been received. The bill passed out of the Senate Governmental Organization Committee on a vote of 14 to 0.

PROPOSED CHANGES TO THE LAW

Existing law:

1. Establishes CDT within the Government Operations Agency (GovOps), under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Gov. Code Sec. 11545(a).)
2. Provides that the duties of the Director include:
3. advising the Governor on the strategic management and direction of the state’s information technology (IT) resources;
4. establishing and enforcing state IT strategic plans, policies, standards, and enterprise architecture, as specified;
5. minimizing overlap, redundancy, and cost in state IT operations by promoting the efficient and effective use of information technology;
6. providing technology direction to agency and department chief information officers to ensure the integration of statewide technology initiatives, compliance with IT policies and standards, and the promotion of the alignment and effective management of IT services;
7. working to improve organizational maturity and capacity in the effective management of IT; and establishing performance management and improvement processes to ensure state IT systems and services are efficient and effective. (Gov. Code § 11545(b).)
8. Expresses the intent of the Legislature that policies and procedures developed by CDT and Department of General Services (DGS) pertaining to the acquisition of IT goods and services provide for all of the following: the expeditious and value-effective acquisition of IT goods and services to satisfy state requirements; the acquisition of IT goods and services within a competitive framework; the delegation of authority by DGS to each state agency that has demonstrated to DGS’s satisfaction the ability to conduct value-effective IT goods and services acquisitions; and the review and resolution of protests submitted by any bidders with respect to any IT goods and services acquisitions. (Pub. Con. Code § 12101.)
9. Requires CDT, on or before September 1, 2024, to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. (Gov’t Code § 11546.45.5(b).)
10. Requires the comprehensive inventory to include a description of all of the following:
    1. any decision the ADS can make or support, the intended benefits of that use, and the alternatives to that use;
    2. the results of any research assessing the efficacy and relative benefits of the uses and alternatives of the ADS described above;
    3. the categories of data and personal information the ADS uses to make its decisions;
    4. the measures in place, if any, to mitigate the risks, including cybersecurity risk and the risk of inaccurate, unfairly discriminatory, or biased decisions, of the ADS, including performance metrics, cybersecurity controls, privacy controls, risk assessments or audits for potential risks, and measures or processes in place to contest an automated decision. (Gov’t Code § 11546.45.5(c).)
11. Requires CDT, on or before January 1, 2025, and annually thereafter, to submit a report, as specified, of the comprehensive inventory to the Assembly Committee on Privacy and Consumer Protection and the Senate Committee on Governmental Organization. This requirement expires on January 1, 2029. (Gov’t Code § 11546.45.5(d).)
12. Defines the following terms:
    1. “Automated decision system” (ADS) means a computational process derived from machine learning, statistical modeling, data analytics, or AI that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. ADS does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
    2. “High-risk automated decision system” means an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.
    3. “State agency” includes every state office, department, division, bureau, the California State University, the Board of Parole Hearings, and specified boards. It does not include the University of California, the Legislature, the judicial branch, or any board, except as provided. (Gov’t Code § 11546.45.5(a).)

This bill:

1. Requires CDT to develop and adopt regulations to create an AI risk management standard consistent with President Biden’s Blueprint for an AI Bill of Rights, the Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology (NIST), and the Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for Inclusive Change.
2. Requires the standard to include the following:
   1. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
   2. Adverse incident monitoring procedures.
   3. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
   4. An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.
3. Requires the standard to also include a detailed risk assessment procedure for procuring ADS that analyzes all of the following:
   1. Organizational and supply chain governance associated with the ADS.
   2. The purpose and use of the ADS.
   3. Any known potential misuses or abuses of the ADS.
   4. An assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS.
   5. The robustness, accuracy, and reliability of the ADS.
   6. The interpretability and explainability of the ADS.
4. Provides, that in developing the standard, CDT shall collaborate with organizations that represent state and local government employees and industry experts, including public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institutions focused on responsible AI procurement, design, and deployment.
5. Requires CDT to adopt regulations. Commencing 6 months after the date on which those regulations are approved and final, a state agency is prohibited from entering into a contract for a high-risk ADS, or any service that utilizes a high-risk ADS, unless the contract includes a clause that does all of the following:
   1. Provides a completed risk assessment of the relevant high-risk ADS.
   2. Requires the state agency or the high-risk ADS vendor, or both, to adhere to appropriate risk controls.
   3. Provides procedures for adverse incident monitoring.
   4. Requires authorization from the state agency before deployment of high-risk ADS upgrades and enhancements.
   5. Requires the state agency or the high-risk ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the high-risk ADS, and information about how to appeal or opt-out of high-risk ADS decisions or outcomes.
6. Defines the relevant terms, including:
   1. “Artificial intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
   2. “Automated decision system” or “ADS” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
   3. “High-risk automated decision system” or “high-risk ADS” means an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.

COMMENTS

1. Frameworks for responsible development and accountability in AI

Owing to recent advances in processing power and the rise of big data, AI’s capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact, and consume media. Since the widespread introduction of AI systems such as ChatGPT, the world has been in awe of the powers of GenAI, which is a type of artificial intelligence that can create new content, such as text, images, code, or music, by learning from existing data. GenAI models can produce realistic and novel artifacts that resemble the data they were trained on, but do not copy it. For example, generative AI can write a poem, draw a picture, or compose a song based on a given prompt or theme.

In ways we may not fully comprehend, AI empowers and encumbers us. It has been used to accelerate productivity, achieve efficiencies, liberate us from drudgery, write our college essay, help us understand and enjoy the world, connect with each other, and live longer, fuller lives. It has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy. For good or ill, its transformative potential seems boundless. With these recent dramatic advances in the capabilities of AI systems, the need for frameworks for accountability and responsible development have become ever more urgent.

In January of 2017, AI researchers, economists, legal scholars, ethicists, and philosophers met in Asilomar, California to discuss principles for managing the responsible development of AI. The collaboration resulted in the Asilomar Principles. Aspirational rather than prescriptive, these 23 principles were intended to initiate and frame a dialogue by providing direction and guidance for policymakers, researchers, and developers. Its endorsers include 1,200 leading experts in the field of AI, including DeepMind founder Demis Hassabis and the late Stephen Hawking.

The Legislature subsequently adopted ACR 215 (Kiley, Ch. 206, Stats. 2018), which added the State of California to that list by endorsing the Asilomar Principles as guiding values for the development of artificial intelligence and related public policy. In broad strokes, those principles aim to do the following:

• Research issues: create beneficial AI; direct funding toward beneficial innovation; maintain constructive and healthy exchanges between AI researchers and policymakers; promote a culture of trust, cooperation, and transparency among researchers and developers of AI; and avoid corner-cutting on safety standards.
• Ethics and values: promote safety, failure transparency, judicial transparency, and responsible innovation; align human values with innovation; protect privacy and liberty; ensure that the benefits and prosperity created by AI are broadly shared; maintain human control over AI; develop AI that supports rather than subverts social and civil processes; and avoid an AI arms race.
• Longer-term issues: avoid assumptions regarding the capabilities of AI; give AI its due attention; and recognize that its risks are potentially catastrophic or existential. [emphasis added]

As directed by the National AI Initiative Act of 2020, NIST developed the AI Risk Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:

Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.

While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.

These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]

AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.

More recently the Biden Administration has published its Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the

design, use, and deployment of AI to protect the rights of the American public:

• Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
• Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
• Data Privacy: You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used. You should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used. Systems should not employ user experience and design decisions that obfuscate user choice or burden users with defaults that are privacy invasive. Consent should only be used to justify collection of data in cases where it can be appropriately and meaningfully given. Any consent requests should be brief, be understandable in plain language, and give you agency over data collection and the specific context of use; current hard-to-understand notice-and-choice practices for broad uses of data should be changed. Enhanced protections and restrictions for data and inferences related to sensitive domains, including health, work, education, criminal justice, and finance, and for data pertaining to youth should put you first. In sensitive domains, your data and related inferences should only be used for necessary functions, and you should be protected by ethical review and use prohibitions. You and your communities should be free from unchecked surveillance; surveillance technologies should be subject to heightened oversight that includes at least pre-deployment assessment of their potential harms and scope limits to protect privacy and civil liberties. Continuous surveillance and monitoring should not be used in education, work, housing, or in other contexts where the use of such surveillance technologies is likely to limit rights, opportunities, or access. Whenever possible, you should have access to reporting that confirms your data decisions have been respected and provides an assessment of the potential impact of surveillance technologies on your rights, opportunities, or access.
• Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
• Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.^[1]

TechEquity, an organization committed to ensuring technology’s evolution benefits everyone equitably, has also laid out their straightforward AI Policy Principles:

• People who are impacted by AI must have agency to shape the technology that dictates their access to critical needs like employment, housing, and healthcare.
• The burden of proof must lie with developers, vendors, and deployers to demonstrate that their tools do not create harm—and regulators, as well as private [individuals], should be empowered to hold them accountable.
• Concentrated power and information asymmetries must be addressed in order to effectively regulate the technology.

Earlier this year, the AI Procurement Lab and the Center for Inclusive Change (woman-owned non-profit and business, respectively) released their own framework, the Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0) “to provide organizations and procurement teams with an essential tool that classifies the risks embedded within each procurement opportunity for the purposes of risk awareness, assessment, measurement, mitigation, treatment, control, monitoring, and management.” They explain the framework and the need for it:

Unfortunately, certain AI use cases can present new and novel risks to organizations (beyond reputational and legal damages). Such use cases include systems designed to deliver critical, sometimes life-altering, decisions in the context of employment, health, education, housing, finance, public assistance, critical infrastructure, essential utilities, law enforcement, immigration, justice, legal services, biometric identification, safety components and other consequential decision systems. Systems developed for these types of use cases are commonly referred to as “high-risk” systems. They pose a high-risk to those impacted either directly or indirectly by the AI system in question in significant or critical ways.

[RMF PAIS 1.0] focuses primarily on risk management for high-risk systems. . . . More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person's safety, civil rights, and/or fundamental human rights and dignity. AI researchers have repeatedly identified a variety of harms that require our attention when deploying such systems in general and especially when deploying high-risk AI systems. The known harms that have been perpetrated by these systems are particularly troubling because they stem from historic systemic bias that can be widely scaled across vulnerable populations through powerful and non-transparent algorithmic computations, generating unfair, unequal, disproportionate, and potentially life-altering outcomes.

Without proper governance and risk mitigation practices, the computational power of high-risk systems can pose unwanted threats to our fellow humans--causing more than just reputation or legal damages for organizations. Unmitigated risks and negative outcomes can, in some cases, mean life or death to the end uses. Hence, the stakes are high, and organizations must find ways to control the risks in order to establish trust for all stakeholders. The RMF PAIS 1.0 provides a guide to identifying and controlling these risks through the use of standard procurement lifecycle processes in a practical and responsible way.^[2]

Seeking to establish a framework for California, Governor Gavin Newsom issued Executive Order N-12-23 “to study the development, use, and risks of artificial intelligence (AI) technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government.”^[3]

The executive order includes the following provisions:

• Risk-Analysis Report: Directs state agencies and departments to perform a joint risk-analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
• Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
• Beneficial Uses of GenAI Report: Direct state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
• Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including CDT-approved environments or “sandboxes” to test such projects.
• State Employee Training: To support California’s state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy, agencies will provide trainings for state government workers to use state-approved GenAI to achieve equitable outcomes, and will establish criteria to evaluate the impact of GenAI to the state government workforce.
• Legislative Engagement: Engage with Legislative partners and key stakeholders, including academic institutions, in a formal process to develop policy recommendations for responsible use of AI, including any guidelines, criteria, reports, and/or training.
• Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.

1. A framework for California: guiding procurement and use of ADS

ADS are algorithm-driven applications that can assist or supplant human decisionmaking processes in areas such as credit decisions, employment screening, insurance eligibility, and the delivery of government services. ADS process enormous datasets and make decisions with speed and reliability that vastly exceed human capabilities. However, poorly designed or poorly understood systems can create unfair, biased, and inaccurate results. When deployed by government agencies, flawed ADS may disproportionately harm low-income families and communities of color and undermine trust in the public sector. Moreover, norms of participatory governance and due process may be jeopardized when ADS affect agency policymaking, adjudications, or enforcement.

Last year, AB 302 (Ward, Ch. 800, Stats. 2023) required CDT, on or before September 1, 2024, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. This bill directs CDT to develop and adopt regulations to create an AI risk management standard that will govern the use of ADS in California. State agencies are to use the framework and attendant regulations to guide their use and procurement of ADS.

The standard must include methods for appropriate risk controls between the relevant state agency and its ADS vendor, adverse incident monitoring procedures, and an analysis about how the use of high-risk ADS can impact vulnerable individuals and communities. The standard is also required to draw a bright line to identify and classify specific use cases and applications for which ADS will not be allowed in state entities.

The bill requires the standard to include a detailed risk assessment procedure for procuring ADS that analyzes all of the following:

• Organizational and supply chain governance associated with the ADS.
• The purpose and use of the ADS.
• Any known potential misuses or abuses of the ADS.
• An assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS.
• The robustness, accuracy, and reliability of the ADS.
• The interpretability and explainability of the ADS.

The bill calls out a number of the frameworks discussed above to guide creation of California’s framework, namely President Biden’s Blueprint for an AI Bill of Rights, the AI RMF 1.0, and the RMF PAIS 1.0.

According to the author:

Artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now. The public has been left vulnerable to the dangers AI poses because of congressional failure to act and the regulatory desert they’ve created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead. We are the home of Silicon Valley, we are leading the way in AI development and innovation and we are also one of the largest purchasing entities of technology in the world. But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values.

1. Stakeholder positions

The Electronic Frontier Foundation writes in support:

SB 892 rightly recognizes the potential risks of AI technology alongside its potential for fostering innovation. It sets the foundation for a framework that reflects principles that are necessary to create an environment that the public can trust, including requirements to:

• Assess the risks associated with procuring high-risk AI systems and ADS, particularly in areas that have a high impact on individual rights, including but not limited to health, education, employment, insurance, utilities, critical infrastructure, public services, and justice/legal.
• Require the development and adoption of risk management standards that include risk assessment, appropriate risk controls, and adverse incident monitoring when procuring such systems.
• Require engagement and consultation with a diversity of stakeholders, including those most affected by government use of these systems, such as public trust and safety experts, community-based organizations, civil society groups, workers, academic researchers and members of the public.
• Ensure people have access to robust notices and opportunities to appeal or opt out of high-risk AI system outcomes or ADS decisions.

Writing in support, the Greenlining Institute states:

This bill, focused on the safe procurement of artificial intelligence (AI) decision systems in the public sector, is necessary to ensure that government agencies do not deploy AI systems that are inaccurate, unfair, or otherwise harm Californians. This accountability is key to building trust and encouraging public adoption of AI technologies. The Greenlining Institute encourages further clarifying that risk assessments should be part of the safety, privacy and non discrimination standards developed by the Department of Technology.

SUPPORT

AI Procurement Lab

American Federation of Musicians

Electronic Frontier Foundation

Greenlining Institute

Secure Justice

Surveillance Resistance Lab

TechEquity Collaborative

OPPOSITION

None received

RELATED LEGISLATION

Pending Legislation:

SB 893 (Padilla, 2024) requires GovOps, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California Artificial Intelligence Research Hub in GovOps, as prescribed. SB 893 requires the hub to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development that seeks to harness the technology’s full potential for public benefit while safeguarding privacy, advancing security, and addressing risks and potential harms to society, as prescribed. SB 893 is currently in this Committee.

SB 896 (Dodd, 2024) largely codifies Governor Newsom’s executive order on the use of GenAI. The bill requires assessments of the beneficial uses, potential harms, and risks to critical infrastructure of GenAI. The bill calls for the development of guidelines for public sector procurement, uses, and required trainings for the use of GenAI. The bill places obligations on state entities with respect to the use of GenAI and ADS. SB 896 is currently in this Committee.

SB 942 (Becker, 2024) establishes the California AI Transparency Act, which, among other things, requires a covered provider, as defined, to create an AI detection tool by which a person can query the covered provider as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a generative AI system, as defined, provided by the covered provider that meets certain criteria. Covered providers are required to include in AI-generated content a visible disclosure that, among other things, includes a clear and conspicuous notice, that identifies the content as generated by AI. SB 942 requires a covered provider to register with CDT and provide them a URL to any AI detection tool it has created. SB 942 is currently in this Committee.

SCR 17 (Dodd, 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems. SCR 17 is currently in the Assembly Privacy and Consumer Protection Committee.

AB 331 (Bauer-Kahan, 2023) prohibits “algorithmic discrimination,” that is, use of an automated decision tool to contribute to unjustified differential treatment or outcomes that may have a significant effect on a person’s life. It requires any deployer of an automated decision tool to perform an impact assessment for those tools and to notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision. AB 331 was held in the Senate Appropriations Committee.

AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. AB 2013 is currently in the Assembly Privacy and Consumer Protection Committee.

AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer and a developer of an automated decision tool to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The assessments must be provided to the Civil Rights Department within 7 days of a request. AB 2930 requires a deployer to, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool.

AB 2930 is currently in the Assembly Privacy and Consumer Protection Committee.

Prior Legislation:

AB 302 (Ward, Ch. 800, Stats. 2023) See Comment 2.

AB 13 (Chau, 2021) would have established the Automated Decision Systems Accountability Act, which, in the context of the State’s procurement policies, promotes oversight over ADS that pose a high risk of adverse impacts on individual rights. The bill was eventually gutted and amended to address a different topic.

SB 444 (Umberg, 2019) would have requested the Regents of the University of California (UC) to enact a resolution authorizing the law schools at UC Berkeley and UC Irvine to participate in a pilot project to develop AI or machine-learning solutions to address access to justice issues faced by self-representing litigants in their respective courts. The bill died in the Assembly Higher Education Committee.

AB 1576 (Calderon, 2019) would have required the Secretary of GovOps to appoint participants to an AI working group to evaluate the uses, risks, benefits, and legal implications associated with the development and deployment of AI by California-based businesses. The bill was held on the Senate Appropriations Committee suspense file.

SJR 6 (Chang, Res. Ch. 112, Stats. 2019) urged the President and the Congress of the United States to develop a comprehensive AI Advisory Committee and to adopt a comprehensive AI policy.

ACR 215 (Kiley, Resolution Ch. 206, Stats. 2018) See Comment 1.

PRIOR VOTES:

Senate Governmental Organization Committee (Ayes 14, Noes 0)

**************

1. Blueprint For An AI Bill Of Rights (October 2022) Office of Science and Technology Policy, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. ↑

2. Dr. Cari L. Miller & Gisele Waters, Ph.D., Risk Management Framework for the Procurement of AI Systems (2024) AI Procurement Lab & The Center for Inclusive Change, https://20596328.fs1.hubspotusercontent-na1.net/hubfs/20596328/Risk%20Management%20Framework%20for%20AI%20Procurement-4.pdf. ↑

3. Press Release, Governor Newsom Signs Executive Order to Prepare California for the Progress of Artificial Intelligence (September 6, 2023) Office of Governor Gavin Newsom, https://www.gov.ca.gov/2023/09/06/governor-newsom-signs-executive-order-to-prepare-california-for-the-progress-of-artificial-intelligence/. ↑

SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION

Senator Bill Dodd

Chair

2023 - 2024 Regular

Bill No: Hearing Date: 4/9/2024

SUBJECT:

DIGEST: This bill requires the California Department of Technology (CDT) to develop and adopt regulations to create an artificial intelligence (AI) risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an automated decision system (ADS) unless the contract includes a clause that provides a complete risk assessment, as specified.

ANALYSIS:

Existing law:

1. Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the CDT.
2. Requires all other contracts for the acquisition of IT goods or services to be made by or under the supervision of the Department of General Services (DGS).
3. The Administrative Procedures Act (APA) governs the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law.
4. Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk ADS that have been proposed for use or are being used, developed, or procured by state agencies, as specified.

This bill:

1. Requires CDT to develop and adopt regulations to create an AI risk management standard and requires the standard to be consistent with all of the following publications:
2. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, published by the White House Office of Science and Technology Policy in October 2022.
3. The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology in January 2023.
4. The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for Inclusive Change in 2024.
5. Requires the AI risk management standard to include all of the following:
6. A detailed risk assessment procedure for procuring ADS that analyzes all of the following:
   • 1. Organizational and supply chain governance associated with the ADS.
     2. The purpose and use of the ADS.
     3. Any known potential misuses or abuses of the ADS.
     4. An assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS.
     5. The robustness, accuracy, and reliability of the ADS.
     6. The interpretability and explainability of the ADS.
7. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
8. Adverse incident monitoring procedures.
9. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
10. An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.
11. Requires the CDT, in order to develop the AI risk management standard, collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researches, and research institutions focused on responsible AI procurement, design, and deployment.
12. Requires the CDT to adopt regulations pursuant to this bill in accordance with the provisions of the APA, as specified.
13. Prohibits, commencing on the date on which the regulations described above are approved and final, a state agency from entering into a contract for an ADS, or any service that utilizes an ADS, unless the contract includes a clause that does all of the following:
    1. Provides a completed risk assessment of the relevant ADS.
    2. Requires the state agency or the ADS vendor, or both, to adhere to appropriate risk controls.
    3. Provides procedures for adverse incident monitoring.
    4. Requires authorization from the state agency before deployment of ADS upgrades and enhancements.
    5. Requires the state agency or the ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, and information about how to appeal or opt-out of ADS decisions or outcomes.

Background

Author Statement. According to the author’s office, “artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now.”

Further, “the public has been left vulnerable to the dangers AI poses because of congressional failure to act and the regulatory desert they’ve created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead. We are the home of Silicon Valley, we are leading the way in AI development and innovation and we are also one of the largest purchasing entities of technology in the world. But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values.”

Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.

Noting California’s role as a global hub for AI and the natural leader in this emerging field of technology, Governor Newsom issued Executive Order (EO) N-12-23 in September of 2023 in order to deploy generative AI ethically and responsibly throughout state government, protect and prepare for potential harms, and remain the world’s AI leader. The EO can generally be broken up into the following elements:

• 1. Risk-Analysis Report: Directs state agencies and departments to perform a joint-risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of generative AI.
  2. Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of generative AI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s (NIST) AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where generative AI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
  3. Beneficial Uses of Generative AI Report: Directs state agencies and departments to develop a report examining the most significant and beneficial uses of generative AI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
  4. Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting generative AI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of generative AI projects, including CDT approved environments or “sandboxes” to test such projects.
  5. Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of generative AI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.

The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.

The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.

Secondly, the Blueprint includes “Algorithmic Discrimination Protections.” Algorithmic discrimination occurs when automated systems contribute to unjustified different treatment or impacts disfavoring people based on their race, color, ethnicity, sex (including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual orientation), religion, age, national origin, disability, veteran status, genetic information, or any other classification protected by law. Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way.

Additionally, the Blueprint discusses “Data Privacy,” stating that individuals should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used.

Further, the Blueprint identifies “Notice and Explanation.” Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.

The final principle identified by the Blueprint is “Human Alternatives, Consideration, and Fallback.” The Blueprint states that you should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts. In some cases, a human or other alternative may be required by law.

National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST was founded in 1901 and is now part of the U.S. Department of Commerce. NIST is one of the nation’s oldest physical science laboratories. Today, NIST measurements support the smallest of technologies to the largest and most complex of human-made creations – from nanoscale devices to tiny that tens of thousands can fit on the end of a single human hair up to earthquake-resistant skyscrapers and global communication networks.

As directed by the National AI Initiative Act of 2020, NIST released the AI RMF in January 2023, to better manage risks to individuals, organizations, and society associated with AI. NIST’s AI RMF is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. The framework was developed through a consensus-driven, open, transparent, and collaborative process that included a Request for Information, several draft versions for public comments, multiple workshops, and other opportunities to provide input. It is intended to build on, align with, and support AI risk management efforts by others. The framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:

1. AI technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high- or low-probability, systemic or localized, and high- or low-impact.
2. While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits- can emerge from an interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
3. These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society.
4. AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by promoting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.

Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). In 2024, the AI Procurement Lab (an independent, woman-owned, non-profit organization that advises government organizations) and the Center for Inclusive Change (a woman-owned small business that contributes AI related research and education programs) released their joint RMF PAIS. The two organizations worked together to create the framework which primarily focuses on risk management for high-risk systems. More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person’s safety, civil rights, and/or fundamental human rights and dignity.

According to the report, “AI researchers have repeatedly identified a variety of harms that require our attention when deploying such systems in general and especially when deploying high-risk AI systems. The known harms that have been perpetrated by these systems are particularly troubling because they stem from historic systemic bias that can be widely scaled across vulnerable populations through powerful and non-transparent algorithmic computations, generating unfair, unequal, disproportionate, and potentially life-altering outcomes.”

California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to move forward with draft ADS technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students. According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”

Further, CalMatters states that under the proposed rules, “businesses must notify people before using AI. If people opt out of interacting with an AI model, businesses cannot discriminate against people for that choice. If people agree to use an AI service or tool, businesses must respond to requests by individuals about how they use their personal information to make predictions. The rules would also require employers or third-party contractors to carry out risk assessments to evaluate the performance of their technology.”

According to the CPPA, the proposed framework includes the following three key components:

1. Pre-use Notice Requirements. A business must provide information to the consumer about how it proposes to use the ADS so that the consumer can decide whether to opt-out or proceed, and whether to exercise their access right.
2. Opt-out Right Requirements. For opt-out requests submitted before the business initiates the processing, the business must not process consumer’s personal information using that ADS. For opt-out requests submitted after the business has initiated the processing, the business must cease processing the consumer’s personal information using that ADS and notify relevant entities of the opt-out and instruct them to comply.
3. Access Right Requirements. This involves the purpose of using the ADS; the output with respect to the consumer; how the businesses used the output with respect to the consumer; how the ADS worked with respect to the consumer; and that the businesses cannot retaliate against consumers for exercising CPPA rights.

The CPPA will spend the rest of the spring and summer performing a statewide informational outreach tour before initiating formal rulemaking, which is expected later this year.

High-Risk Automated Decision Systems. AB 302 (Ward, Chapter 800, Statutes of 2023) directed CDT, in coordination with other interagency bodies as it deems appropriate, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. Per the bill, that inventory must be completed on or before September 1, 2024.

Further, AB 302 (Ward, Chapter 800, Statutes of 2023) required the inventory to include: any decision the ADS can make or support and the intended benefits of that use and the alternatives to any use; the results of any research assessing the efficacy and relative benefits of the uses and alternatives of the ADS; the categories of data and personal information the ADS uses to make its decisions; and the measures in place, if any, to mitigate risks including but not limited to performance metrics, cybersecurity controls, privacy controls, risk assessments or audits for potential risks, and measures or processes in place to contest an automated decision.

AB 302 (Ward, Chapter 800, Statutes of 2023) defined “high-risk ADS” to mean an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.

Administrative Procedures Act and California’s Regulatory Processes. The Office of Administrative Law (OAL) is responsible for ensuring that California state agencies comply with the rulemaking procedures and standards set forth in California’s Administrative Procedures Act (APA). A “regulation” is any rule, regulation, order or standard of general application or the amendment, supplement, or revision of any rule, regulation, order, or standard adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it.

When adopting regulations, every department, division, office, officer, bureau, board or commission in the executive branch of the California state government must follow the rulemaking procedures in the APA and regulations adopted by the OAL, unless expressly exempted by statute from some or all of these requirements.

The APA requirements are designed to provide the public with a meaningful opportunity to participate in the adoption of regulations or rules that have the force of law by California state agencies and to ensure the creation of an adequate record for the OAL and judicial review.

Regulations subject to the APA are generally adopted through the “Regular” or “Emergency” rulemaking processes. The rulemaking process used by an agency to adopt regulations will dictate what procedural requirements must be followed, including but not limited to the contents of the rulemaking record, timeframes, opportunities for public participation, OAL’s review, and effective dates for the regulations.

AI Risk Management Standard. This bill requires the CDT to develop and adopt regulations to create an AI risk management standard. Specifically, this bill requires the standard developed by CDT to be consistent with three identified publications: the White House’s Blueprint; NIST’s AI RMF 1.0; and the AI Procurement Lab and the Center for Inclusive Change co-produced RMF PAIS 1.0 – each discussed above.

This bill requires the standard developed by CDT to include all five of the following:

1. A detailed risk assessment procedure for procuring ADS that analyzes each of the following: organizational and supply chain governance associated with ADS; the purpose and use of the ADS; any known potential misuses or abuses of the ADS; an assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS; the robustness, accuracy, and reliability of the ADS; the interpretability and explainability of the ADS.
2. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
3. Adverse incident monitoring procedures.
4. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
5. An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.

To develop the standard, this bill requires CDT to collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institution focused on responsible AI procurement, design, and deployment. This bill specifies that CDT must adopt the regulations required by this measure in accordance with the provisions of the APA.

Once the regulations have been approved and final, this bill prohibits any state agency from entering into a contract for an ADS system, or any service that utilizes an ADS, unless the contract includes a clause that does all of the following:

1. Provides a completed risk assessment of the relevant ADS.
2. Requires the state agency or the ADS vendor, or both, to adhere to appropriate risk controls.
3. Provides procedures for adverse incident monitoring.
4. Requires authorization from the state agency before deployment of ADS upgrades and enhancements.
5. Requires the state agency or the ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, and information about how to appeal or opt-out of ADS decisions or outcomes.

Suggested Amendments. In an attempt to align the myriad bills relating to AI this year around a single definition of what AI is, bills in Senate policy committees have been adopting the following definition:

Amendment #1 “Artificial Intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.

This is different than the definition currently in this bill, therefore the Committee may wish to amend this bill to instead utilize the above definition of AI.

Additionally, this bill requires agencies and vendors to comply with the new regulations immediately, once they are approved and final, which may become problematic if agencies or vendors are not prepared for the new regulations.

To allow for agencies and vendors to adapt and comply with the future regulations, the Committee may wish to amend the bill as follows, allowing for a six month grace period for implementation:

Amendment #2 (c) Commencing 6 months after the date on which the regulations described in subdivision (b) are approved and final, a state agency shall not enter into a contract for automated decision system, or any service that utilizes automated decision system, unless the contract includes a clause that does all of the following:

Prior/Related Legislation

SB 893 (Padilla, 2024) requires the establishment of the California AI Research Hub in the Government Operations Agency to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development that seeks to harness the technology’s full potential for public benefit, as specified. (Pending in the Senate Government Organization Committee)

SB 896 (Dodd, 2024) the AI Accountability Act, among other things, requires specified state entities to produce a State of California Benefits and Risk of Generative AI Report, as specified; requires specified state entities to perform a joint risk analysis of potential threats posed by the use of generative AI to California’s critical energy infrastructure. (Pending in the Senate Governmental Organization Committee)

SB 942 (Becker, 2024) the California AI Transparency Act, among other things, requires a covered provider to create an AI detection tool by which a person can query the covered provide as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a GenAI system, as specified. (Pending in the Senate Judiciary Committee)

SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, this bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. (Pending in the Senate Governmental Organization Committee)

AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)

AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer, as defined, and a developer of an automated decision tool, to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. (Pending in the Assembly Privacy and Consumer Protection Committee)

AB 3058 (Low, 2024) establishes the California Unconditional Benefit Income Pilot Program for the purpose of providing assistance to individuals who are unemployed because of automation or AI, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)

SB 313 (Dodd, 2023) would have established the Office of Artificial Intelligence in state government and require state agencies to disclose when they are using generative AI to communicate with a person and to provide them an option to speak with a natural person at the agency. (Held on the Senate Appropriations Committee Suspense File)

SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights,” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.

AB 302 (Ward, Chapter 800, Statutes of 2023) requires CDT, in coordination with other interagency bodies, to conduct a comprehensive inventory of all high-risk ADS used by state agencies on or before September 1, 2024, and report the findings to the Legislature by January 1, 2025, and annually thereafter, as specified.

FISCAL EFFECT: Appropriation: Fiscal Com.: Local:

SUPPORT:

AI Procurement Lab

American Federation of Musicians

Center for Inclusive Change

Consumer Reports

Electronic Frontier Foundation

Greenlining Institute

Secure Justice

Surveillance Resistance Lab

TechEquity Collaborative

OPPOSITION:

None received

ARGUMENTS IN SUPPORT: In support of the bill, the AI Procurement Lab writes that, “AI, procured and deployed responsibly, has the potential to revolutionize the delivery of government services to Californians. However, this technology can also pose a significant risk to the safety, privacy, and civil rights of Californians. To protect and build trust, those procuring and using AI systems have a responsibility to add benefit and diligent risk analysis prior to procurement with exacting standards in risk management unique to the AI category. This is especially true when government officials sign contracts for administrative processes that perform tasks displacing their previously exercised discretion. An AI risk management standard is also necessary when front-end government employees depend on the opaque logic of these systems that bears little or no resemblance to the reasoning process of agency personnel.”

Further, the Center for Inclusive Change writes that, “California must ensure that innovation does not come at the expense of Californians. Given the state’s purchasing power through our procurement of technology and services, California has the opportunity to set a national standard that drives the innovation of AI technology and services by requiring that any technology and services contracted by the state adhere to standards that protect Californians. While the Governor’s Executive Order to Prepare California for the Progress of Artificial Intelligence took some early steps to guidelines for public sector procurement, California must take further steps to safeguard the critical services we provide.”

DUAL REFERRAL: Senate Governmental Organization Committee & Senate Judiciary Committee

SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION

Senator Bill Dodd

Chair

2023 - 2024 Regular

Bill No: SB 892 Hearing Date: 4/9/2024

SUBJECT: Public contracts: automated decision systems: AI risk management standards

DIGEST: This bill requires the California Department of Technology (CDT) to develop and adopt regulations to create an artificial intelligence (AI) risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an automated decision system (ADS) unless the contract includes a clause that provides a complete risk assessment, as specified.

ANALYSIS:

Existing law:

1. Requires all contracts for the acquisition of information technology (IT) goods and services related to IT projects to be made by or under the supervision of the CDT.
2. Requires all other contracts for the acquisition of IT goods or services to be made by or under the supervision of the Department of General Services (DGS).
3. The Administrative Procedures Act (APA) governs the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law.
4. Requires CDT, in coordination with other interagency bodies, to conduct, on or before September 1, 2024, and annually thereafter, a comprehensive inventory of all high-risk ADS that have been proposed for use or are being used, developed, or procured by state agencies, as specified.

This bill:

1. Requires CDT to develop and adopt regulations to create an AI risk management standard and requires the standard to be consistent with all of the following publications:
2. The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People, published by the White House Office of Science and Technology Policy in October 2022.
3. The Artificial Intelligence Risk Management Framework (AI RMF 1.0), released by the National Institute of Standards and Technology in January 2023.
4. The Risk Management Framework for the Procurement of Artificial Intelligence (RMF PAIS 1.0), authored by the AI Procurement Lab and the Center for Inclusive Change in 2024.
5. Requires the AI risk management standard to include all of the following:
6. A detailed risk assessment procedure for procuring ADS that analyzes all of the following:
   • 1. Organizational and supply chain governance associated with the ADS.
     2. The purpose and use of the ADS.
     3. Any known potential misuses or abuses of the ADS.
     4. An assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS.
     5. The robustness, accuracy, and reliability of the ADS.
     6. The interpretability and explainability of the ADS.
7. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
8. Adverse incident monitoring procedures.
9. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
10. An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.
11. Requires the CDT, in order to develop the AI risk management standard, collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researches, and research institutions focused on responsible AI procurement, design, and deployment.
12. Requires the CDT to adopt regulations pursuant to this bill in accordance with the provisions of the APA, as specified.
13. Prohibits, commencing on the date on which the regulations described above are approved and final, a state agency from entering into a contract for an ADS, or any service that utilizes an ADS, unless the contract includes a clause that does all of the following:
    1. Provides a completed risk assessment of the relevant ADS.
    2. Requires the state agency or the ADS vendor, or both, to adhere to appropriate risk controls.
    3. Provides procedures for adverse incident monitoring.
    4. Requires authorization from the state agency before deployment of ADS upgrades and enhancements.
    5. Requires the state agency or the ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, and information about how to appeal or opt-out of ADS decisions or outcomes.

Background

Author Statement. According to the author’s office, “artificial intelligence stands to have the largest influence on society since the dawn of the Digital Age. It has the potential to provide incredible societal benefits if harnessed appropriately, but threatens to pose terrible consequences if safeguards are not put in place as it becomes integrated into everyday life. The research and guardrails around generative AI services will become the standard that guides the technology as it proliferates throughout every sector of our economy. The rapid growth of this technology’s capability over even just the past year is clear warning, we must set these safety parameters now.”

Further, “the public has been left vulnerable to the dangers AI poses because of congressional failure to act and the regulatory desert they’ve created. We cannot wait for Congress to overcome their dysfunction, so California must step in and step up to lead. We are the home of Silicon Valley, we are leading the way in AI development and innovation and we are also one of the largest purchasing entities of technology in the world. But, as companies develop the cutting edge of this technology, we must be sure the guardrails around its use are built with California values.”

Navigating the Artificial Intelligence Landscape. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions. The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.

Noting California’s role as a global hub for AI and the natural leader in this emerging field of technology, Governor Newsom issued Executive Order (EO) N-12-23 in September of 2023 in order to deploy generative AI ethically and responsibly throughout state government, protect and prepare for potential harms, and remain the world’s AI leader. The EO can generally be broken up into the following elements:

• 1. Risk-Analysis Report: Directs state agencies and departments to perform a joint-risk analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of generative AI.
  2. Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of generative AI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s (NIST) AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where generative AI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
  3. Beneficial Uses of Generative AI Report: Directs state agencies and departments to develop a report examining the most significant and beneficial uses of generative AI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
  4. Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting generative AI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of generative AI projects, including CDT approved environments or “sandboxes” to test such projects.
  5. Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of generative AI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.

The Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People. In October of 2022, the White House’s Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights (Blueprint) identifying five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of AI. The Blueprint is intended to be a guide for a society to protect all people from AI-related threats.

The five identified principles include “Safe and Effective Systems.” The Blueprint provides that automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system. Systems should undergo pre-deployment testing, risk identification and mitigation, and ongoing monitoring that demonstrate they are safe and effective based on their intended use, mitigation of unsafe outcomes including those beyond the intended use, and adherence to domain-specific standards. Outcomes of these protective measures should include the possibility of not deploying the system or removing a system from use.

Secondly, the Blueprint includes “Algorithmic Discrimination Protections.” Algorithmic discrimination occurs when automated systems contribute to unjustified different treatment or impacts disfavoring people based on their race, color, ethnicity, sex (including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual orientation), religion, age, national origin, disability, veteran status, genetic information, or any other classification protected by law. Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way.

Additionally, the Blueprint discusses “Data Privacy,” stating that individuals should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used.

Further, the Blueprint identifies “Notice and Explanation.” Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.

The final principle identified by the Blueprint is “Human Alternatives, Consideration, and Fallback.” The Blueprint states that you should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts. In some cases, a human or other alternative may be required by law.

National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST was founded in 1901 and is now part of the U.S. Department of Commerce. NIST is one of the nation’s oldest physical science laboratories. Today, NIST measurements support the smallest of technologies to the largest and most complex of human-made creations – from nanoscale devices to tiny that tens of thousands can fit on the end of a single human hair up to earthquake-resistant skyscrapers and global communication networks.

As directed by the National AI Initiative Act of 2020, NIST released the AI RMF in January 2023, to better manage risks to individuals, organizations, and society associated with AI. NIST’s AI RMF is intended for voluntary use and to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems. The framework was developed through a consensus-driven, open, transparent, and collaborative process that included a Request for Information, several draft versions for public comments, multiple workshops, and other opportunities to provide input. It is intended to build on, align with, and support AI risk management efforts by others. The framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:

1. AI technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high- or low-probability, systemic or localized, and high- or low-impact.
2. While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits- can emerge from an interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
3. These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society.
4. AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by promoting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.

Risk Management Framework for the Procurement of AI Systems (RMF PAIS 1.0). In 2024, the AI Procurement Lab (an independent, woman-owned, non-profit organization that advises government organizations) and the Center for Inclusive Change (a woman-owned small business that contributes AI related research and education programs) released their joint RMF PAIS. The two organizations worked together to create the framework which primarily focuses on risk management for high-risk systems. More specifically, given that high-risk systems can produce great advantages in terms of efficiency gains and consistency in decision-making output, they also have the potential to impact a person’s safety, civil rights, and/or fundamental human rights and dignity.

According to the report, “AI researchers have repeatedly identified a variety of harms that require our attention when deploying such systems in general and especially when deploying high-risk AI systems. The known harms that have been perpetrated by these systems are particularly troubling because they stem from historic systemic bias that can be widely scaled across vulnerable populations through powerful and non-transparent algorithmic computations, generating unfair, unequal, disproportionate, and potentially life-altering outcomes.”

California Privacy Protection Agency Actions. In March of this year, the California Privacy Protection Agency (CPPA) voted to move forward with draft ADS technology and risk assessment regulations regarding how businesses use AI and collect the personal information of consumers, workers, and students. According to a March 13 article by CalMatters, “[t]he proposed rules seek to create guidelines for the many areas in which AI and personal data can influence the lives of Californians: job compensation, demotion, and opportunity; housing, insurance, health care, and student expulsion. For example, under the rules, if an employer wanted to use AI to make predictions about a person’s emotional state or personality during a job interview, a job candidate could opt out without fear of discrimination for choosing to do so.”

Further, CalMatters states that under the proposed rules, “businesses must notify people before using AI. If people opt out of interacting with an AI model, businesses cannot discriminate against people for that choice. If people agree to use an AI service or tool, businesses must respond to requests by individuals about how they use their personal information to make predictions. The rules would also require employers or third-party contractors to carry out risk assessments to evaluate the performance of their technology.”

According to the CPPA, the proposed framework includes the following three key components:

1. Pre-use Notice Requirements. A business must provide information to the consumer about how it proposes to use the ADS so that the consumer can decide whether to opt-out or proceed, and whether to exercise their access right.
2. Opt-out Right Requirements. For opt-out requests submitted before the business initiates the processing, the business must not process consumer’s personal information using that ADS. For opt-out requests submitted after the business has initiated the processing, the business must cease processing the consumer’s personal information using that ADS and notify relevant entities of the opt-out and instruct them to comply.
3. Access Right Requirements. This involves the purpose of using the ADS; the output with respect to the consumer; how the businesses used the output with respect to the consumer; how the ADS worked with respect to the consumer; and that the businesses cannot retaliate against consumers for exercising CPPA rights.

The CPPA will spend the rest of the spring and summer performing a statewide informational outreach tour before initiating formal rulemaking, which is expected later this year.

High-Risk Automated Decision Systems. AB 302 (Ward, Chapter 800, Statutes of 2023) directed CDT, in coordination with other interagency bodies as it deems appropriate, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. Per the bill, that inventory must be completed on or before September 1, 2024.

Further, AB 302 (Ward, Chapter 800, Statutes of 2023) required the inventory to include: any decision the ADS can make or support and the intended benefits of that use and the alternatives to any use; the results of any research assessing the efficacy and relative benefits of the uses and alternatives of the ADS; the categories of data and personal information the ADS uses to make its decisions; and the measures in place, if any, to mitigate risks including but not limited to performance metrics, cybersecurity controls, privacy controls, risk assessments or audits for potential risks, and measures or processes in place to contest an automated decision.

AB 302 (Ward, Chapter 800, Statutes of 2023) defined “high-risk ADS” to mean an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.

Administrative Procedures Act and California’s Regulatory Processes. The Office of Administrative Law (OAL) is responsible for ensuring that California state agencies comply with the rulemaking procedures and standards set forth in California’s Administrative Procedures Act (APA). A “regulation” is any rule, regulation, order or standard of general application or the amendment, supplement, or revision of any rule, regulation, order, or standard adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it.

When adopting regulations, every department, division, office, officer, bureau, board or commission in the executive branch of the California state government must follow the rulemaking procedures in the APA and regulations adopted by the OAL, unless expressly exempted by statute from some or all of these requirements.

The APA requirements are designed to provide the public with a meaningful opportunity to participate in the adoption of regulations or rules that have the force of law by California state agencies and to ensure the creation of an adequate record for the OAL and judicial review.

Regulations subject to the APA are generally adopted through the “Regular” or “Emergency” rulemaking processes. The rulemaking process used by an agency to adopt regulations will dictate what procedural requirements must be followed, including but not limited to the contents of the rulemaking record, timeframes, opportunities for public participation, OAL’s review, and effective dates for the regulations.

AI Risk Management Standard. This bill requires the CDT to develop and adopt regulations to create an AI risk management standard. Specifically, this bill requires the standard developed by CDT to be consistent with three identified publications: the White House’s Blueprint; NIST’s AI RMF 1.0; and the AI Procurement Lab and the Center for Inclusive Change co-produced RMF PAIS 1.0 – each discussed above.

This bill requires the standard developed by CDT to include all five of the following:

1. A detailed risk assessment procedure for procuring ADS that analyzes each of the following: organizational and supply chain governance associated with ADS; the purpose and use of the ADS; any known potential misuses or abuses of the ADS; an assessment of the legality, traceability, and provenance of the data the ADS uses and the legality of the output of the ADS; the robustness, accuracy, and reliability of the ADS; the interpretability and explainability of the ADS.
2. Methods for appropriate risk controls between the state agency and ADS vendor, including, but not limited to, reducing the risk through various mitigation strategies, eliminating the risk, or sharing the risk.
3. Adverse incident monitoring procedures.
4. Identification and classification of prohibited use cases and applications of ADS that the state shall not procure.
5. An analysis about how the use of high-risk ADS can impact vulnerable individuals and communities.

To develop the standard, this bill requires CDT to collaborate with organizations that represent state and local government employees and industry experts, including, but not limited to, public trust and safety experts, community-based organizations, civil society groups, academic researchers, and research institution focused on responsible AI procurement, design, and deployment. This bill specifies that CDT must adopt the regulations required by this measure in accordance with the provisions of the APA.

Once the regulations have been approved and final, this bill prohibits any state agency from entering into a contract for an ADS system, or any service that utilizes an ADS, unless the contract includes a clause that does all of the following:

1. Provides a completed risk assessment of the relevant ADS.
2. Requires the state agency or the ADS vendor, or both, to adhere to appropriate risk controls.
3. Provides procedures for adverse incident monitoring.
4. Requires authorization from the state agency before deployment of ADS upgrades and enhancements.
5. Requires the state agency or the ADS vendor, or both, to provide notice to individuals that would likely be affected by the decisions or outcomes of the ADS, and information about how to appeal or opt-out of ADS decisions or outcomes.

Suggested Amendments. In an attempt to align the myriad bills relating to AI this year around a single definition of what AI is, bills in Senate policy committees have been adopting the following definition:

Amendment #1 “Artificial Intelligence” means an engineered or machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.

This is different than the definition currently in this bill, therefore the Committee may wish to amend this bill to instead utilize the above definition of AI.

Additionally, this bill requires agencies and vendors to comply with the new regulations immediately, once they are approved and final, which may become problematic if agencies or vendors are not prepared for the new regulations.

To allow for agencies and vendors to adapt and comply with the future regulations, the Committee may wish to amend the bill as follows, allowing for a six month grace period for implementation:

Amendment #2 (c) Commencing 6 months after the date on which the regulations described in subdivision (b) are approved and final, a state agency shall not enter into a contract for a high-risk automated decision system, or any service that utilizes a high-risk automated decision system, unless the contract includes a clause that does all of the following:

Prior/Related Legislation

SB 893 (Padilla, 2024) requires the establishment of the California AI Research Hub in the Government Operations Agency to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI research and development that seeks to harness the technology’s full potential for public benefit, as specified. (Pending in the Senate Government Organization Committee)

SB 896 (Dodd, 2024) the AI Accountability Act, among other things, requires specified state entities to produce a State of California Benefits and Risk of Generative AI Report, as specified; requires specified state entities to perform a joint risk analysis of potential threats posed by the use of generative AI to California’s critical energy infrastructure. (Pending in the Senate Governmental Organization Committee)

SB 942 (Becker, 2024) the California AI Transparency Act, among other things, requires a covered provider to create an AI detection tool by which a person can query the covered provide as to the extent to which text, image, video, audio, or multimedia content was created, in whole or in part, by a GenAI system, as specified. (Pending in the Senate Judiciary Committee)

SB 1047 (Wiener, 2024) requires developers of powerful AI models, and those providing the computer power to train such models, to put appropriate safeguards and policies into place to prevent critical harms. Additionally, this bill establishes a state entity to oversee the development of these models and calls for the creation of a public cloud computer cluster known as CalCompute, as specified. (Pending in the Senate Governmental Organization Committee)

AB 2013 (Irwin, 2024) requires, on or before January 1, 2026, a developer, as defined, of an AI system or service to post on the developer’s website documentation regarding the data used to train the AI system or service, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)

AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer, as defined, and a developer of an automated decision tool, to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. (Pending in the Assembly Privacy and Consumer Protection Committee)

AB 3058 (Low, 2024) establishes the California Unconditional Benefit Income Pilot Program for the purpose of providing assistance to individuals who are unemployed because of automation or AI, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)

SB 313 (Dodd, 2023) would have established the Office of Artificial Intelligence in state government and require state agencies to disclose when they are using generative AI to communicate with a person and to provide them an option to speak with a natural person at the agency. (Held on the Senate Appropriations Committee Suspense File)

SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights,” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.

AB 302 (Ward, Chapter 800, Statutes of 2023) requires CDT, in coordination with other interagency bodies, to conduct a comprehensive inventory of all high-risk ADS used by state agencies on or before September 1, 2024, and report the findings to the Legislature by January 1, 2025, and annually thereafter, as specified.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

SUPPORT:

AI Procurement Lab

American Federation of Musicians

Center for Inclusive Change

Consumer Reports

Electronic Frontier Foundation

Greenlining Institute

Secure Justice

Surveillance Resistance Lab

TechEquity Collaborative

OPPOSITION:

None received

ARGUMENTS IN SUPPORT: In support of the bill, the AI Procurement Lab writes that, “AI, procured and deployed responsibly, has the potential to revolutionize the delivery of government services to Californians. However, this technology can also pose a significant risk to the safety, privacy, and civil rights of Californians. To protect and build trust, those procuring and using AI systems have a responsibility to add benefit and diligent risk analysis prior to procurement with exacting standards in risk management unique to the AI category. This is especially true when government officials sign contracts for administrative processes that perform tasks displacing their previously exercised discretion. An AI risk management standard is also necessary when front-end government employees depend on the opaque logic of these systems that bears little or no resemblance to the reasoning process of agency personnel.”

Further, the Center for Inclusive Change writes that, “California must ensure that innovation does not come at the expense of Californians. Given the state’s purchasing power through our procurement of technology and services, California has the opportunity to set a national standard that drives the innovation of AI technology and services by requiring that any technology and services contracted by the state adhere to standards that protect Californians. While the Governor’s Executive Order to Prepare California for the Progress of Artificial Intelligence took some early steps to guidelines for public sector procurement, California must take further steps to safeguard the critical services we provide.”

DUAL REFERRAL: Senate Governmental Organization Committee & Senate Judiciary Committee