AB 1018: Automated decision systems.
- Session Year: 2025-2026
- House: Assembly
- Latest Version Date: 2026-08-28
Current Status:
In Progress
(2026-08-30: Read second time. Ordered to third reading.)
Introduced
In Committee
Awaiting First Chamber Vote
Passed First Chamber
In Committee
Awaiting Second Chamber Vote
Passed Second Chamber
Enacted
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including providing a potential deployer instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
This bill would require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS that results in an adverse outcome, as defined, and provide the subject with an opportunity to appeal the outcome of the consequential decision, as prescribed.
This bill would authorize certain public entities, including the Attorney General, General to bring a specified civil action for noncompliance.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct impact assessments of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including a high-level summary of the results of those impact assessments. providing a potential deployer instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
This bill would, beginning January 1, 2027, would require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS that results in an adverse outcome, as defined, and provide the subject with an opportunity to appeal the outcome of the consequential decision, as prescribed.
This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.This bill would make legislative findings to that effect.The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct performance evaluations impact assessments of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including a high-level summary of the results of those performance evaluations. impact assessments.
This bill would, beginning January 1, 2027, require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS and provide the subject with an opportunity to appeal the outcome of the consequential decision, as prescribed.
This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the performance evaluation impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.
This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct performance evaluations of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including the results of those performance evaluations.
This bill would, beginning January 1, 2027, require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS, provide the subject an opportunity to opt out of the use of the covered ADS, ADS and provide the subject with an opportunity to appeal the outcome of the consequential decision, and submit the covered ADS to third-party audits, as prescribed. The bill would also prescribe requirements for a third party to audit a covered ADS.
This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the performance evaluation or disparate impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.
This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct performance evaluations of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including the results of those performance evaluations.
This bill would, beginning January 1, 2027, require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS, provide the subject an opportunity to opt out of the use of the covered ADS, provide the subject with an opportunity to appeal the outcome of the consequential decision, and submit the covered ADS to third-party audits, as prescribed. The bill would also prescribe requirements for a third party to audit a covered ADS.
This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the performance evaluation or disparate impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.
This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct performance evaluations of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including the results of those performance evaluations.
This bill would, beginning January 1, 2027, require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS, provide the subject an opportunity to opt out of the use of the covered ADS, provide the subject with an opportunity to appeal the outcome of the consequential decision, and submit the covered ADS to third-party audits, as prescribed. The bill would also prescribe requirements for a third party to audit a covered ADS.
This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the performance evaluation or disparate impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.
This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct performance evaluations of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including the results of those performance evaluations.
This bill would, beginning January 1, 2027, require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS, provide the subject an opportunity to opt out of the use of the covered ADS, provide the subject with an opportunity to appeal the outcome of the consequential decision, and submit the covered ADS to third-party audits, as prescribed. The bill would also prescribe requirements for a third party to audit a covered ADS.
This bill would prescribe requirements for a third party to audit a covered ADS, as prescribed.This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the performance evaluation or disparate impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.
This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law requires, on or before September 1, 2024, the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
This bill would generally regulate the development and deployment of an automated decision system (ADS) used to make consequential decisions, as defined. The bill would define automated decision system to mean a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
This bill would require a developer of a covered ADS, as defined, to take certain actions, including conduct performance evaluations of the covered ADS and provide deployers to whom the developer transfers the covered ADS with certain information, including the results of those performance evaluations.
This bill would, beginning January 1, 2027, require a deployer of a covered ADS to take certain actions, including provide certain disclosures to a subject of a consequential decision made or facilitated by the covered ADS, provide the subject an opportunity to opt out of the use of the covered ADS, provide the subject with an opportunity to appeal the outcome of the consequential decision, and submit the covered ADS to third-party audits, as prescribed.
This bill would prescribe requirements for a third party to audit a covered ADS, as prescribed.
This bill would require a developer, deployer, or auditor to, within 30 days of receiving a request from the Attorney General, provide an unredacted copy of the performance evaluation or disparate impact assessment prepared pursuant to the bill to the Attorney General and would exempt those records from the California Public Records Act.
This bill would authorize certain public entities, including the Attorney General, to bring a specified civil action for noncompliance.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
As used in this chapter:
(a)Adverse outcome means either of the following:
(1)A decision that denies, terminates, revokes, or materially reduces or restricts a natural persons access to, eligibility for, selection for, compensation for, or the provision of an opportunity or service.
(2)A decision that results in materially less favorable differentiated price, cost, compensation, or other material terms that are reasonably likely to materially limit, delay, or effectively deny, or otherwise fundamentally alter, a natural persons access to, eligibility for, selection for, compensation for, or the provision of an opportunity or service compared to terms offered to similarly situated natural persons. If a decision outcome imposes materially less favorable differentiated pricing or terms, the decision outcome materially impacts price, cost sharing, compensation, or material terms.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not include a spam email filter, firewall, antivirus software, identity verification or access management tool, calculator, database, dataset, spreadsheet, document validation or authenticity tool, or other compilation of data or information.
(d)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person: decision, determination, or action made about a natural person that relates to the provision of or a natural persons access to, eligibility for, selection for, or compensation for any of the following:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Accreditation.
(C)Certification.
(D)Admissions and enrollment.
(E)Discipline.
(F)Financial aid and scholarships.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Health care and health insurance, including mental health care, dental, and vision.
(5)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(6)Access to government benefits or services.
(7)Insurance.
(e)(1)Covered ADS means an automated decision system that is designed for, or used by, a deployer to profile a natural person in order for the deployer to make or facilitate a consequential decision.
(2)Covered ADS does not mean a GenAI system that is not intended or advertised to be used by a deployer to profile a natural person in order to make or facilitate a consequential decision.
(f)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(g)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
(h)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision.
(i)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine-tuning of the covered ADS.
(j)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(k)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(l)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(m)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is both of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(2)Express consent does not include an authorization that is any of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(C)Contained within a more general notice, agreement, or set of terms and conditions.
(n)FERPA means the federal Family Educational Rights and Privacy Act of 1974 (20 U.S.C. Sec. 1232g et seq.).
(o)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(p)GenAI system means an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio, that emulates the structure and characteristics of the systems training data.
(q)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.(r)(q)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(s)(r)Profile means to use a covered ADS that does both of the following:
(1)Receives, as input data, the personal information of a subject of the consequential decision.
(2)Issues a simplified output, including a score, classification, or recommendation, in order to evaluate the subject or predict outcomes specific to the subject.
(t)(s)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(u)(t)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not include a modification that results from fine-tuning.
(v)(u)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
(w)(v)The definitions of disparate impact and disparate treatment provided in subdivisions (i) and (j) are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the California Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
22756.1.(a)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(1)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(2)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(3)Any technical information necessary for the deployer to comply with this chapter. chapter, including any changes to the information provided under paragraphs (1) and (2) if a substantial modification is made to the covered ADS.
(b)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(1)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(2)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(3)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(c)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain both of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus five years:
(1)Any documentation provided to deployers pursuant to this chapter.
(2)Records of any redactions made pursuant to this chapter.
(d)This section does not apply to a developer subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) with respect to a covered ADS that is intended to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
22756.2.(a)(1)Except as provided in paragraph (2), before a deployer finalizes a consequential decision made or facilitated by a covered ADS used by the deployer, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information: clear and conspicuous notice to a subject that the deployer used, or will use, a covered ADS in a consequential decision affecting the subject and instructions regarding how the subject may obtain the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.(B)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.(C)(A)The types, categories, and sources of personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(D)The subjects rights under subdivision (c) and the means and timeframe for exercising those rights.(E)(B)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(3)The disclosure made pursuant to this subdivision may be made through a generalized notice on the deployers internet website or by using an automatic reply mechanism.
(b)(1)If a deployer finalizes a consequential decision made or facilitated by a covered ADS that results in an adverse outcome, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within thirty 30 days:
(A)A plain language description of the consequential decision and the role the covered ADS played in the consequential decision.
(B)Instructions and a simple-to-follow process to request the following information:
(A)(i)The personal characteristics or attributes of the subject that the covered ADS measured or assessed to make or facilitate the consequential decision.
(B)(ii)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)(iii)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)(iv)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)(v)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)(vi)Whether a natural person reviewed either of the following before the consequential decision was finalized:
(i)(I)The outputs of the covered ADS.
(ii)(II)The outcome of the consequential decision.
(G)(vii)Contact information for the deployer.
(H)(viii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(I)(ix)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(J)(x)The subjects rights under subdivision (c) and the means and timeframe for exercising those rights.
(2)Paragraph (1) does not apply to either of the following:
(A)A medical emergency.
(B)The sorting by a covered ADS of resumes based on bona fide occupational qualifications.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(4)A deployer that receives a request for any of the information pursuant to subparagraph (B) of paragraph (1) shall provide the information within 30 business days of the request.
(c)If a deployer that employs more than 25 employees finalizes a consequential decision made or facilitated by a covered ADS that results in an adverse outcome, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall respond to a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision, if feasible.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that determines that rectifying the decision is infeasible shall do both of the following:
(i)Provide the subject with an explanation of the basis for the determination.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(d)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision may use existing forms of providing that notice to comply with this section and need not duplicatively provide notice to the subject under this section.
(3)For a consequential decision relating to education, a deployer that is subject to FERPA satisfies the notice and disclosure requirements of this section by providing notice and disclosures through processes and channels that are consistent with FERPA and the deployers FERPA notices and student record access procedures, including, if applicable, notice to a parent or guardian or an eligible student.
(4)A deployer that is subject to FERPA is not required to provide a separate or duplicative notice or disclosure process if the deployer has established a notice or disclosure process to comply with FERPA.
(5)A deployer that is subject to FERPA is not required to establish a separate or duplicative correction and reconsideration process if the deployer has established a correction and reconsideration process to comply with FERPA.
(6)(A)A creditor, with respect to a consequential decision involving the offering, denial, pricing, servicing, or other material terms of credit that provides a notice to a consumer pursuant to the federal Equal Credit Opportunity Act (15 U.S.C. Sec. 1691 et seq.) and, if applicable, the federal Fair Credit Reporting Act (15 U.S.C. Sec. 1681 et seq.), shall be deemed in compliance with the notice or disclosure requirements of Section 22756.2 that relate to the same decision if the notice provided to the consumer pursuant to those laws also satisfies the notice or disclosure requirements of this section.
(B)If a creditor complies with applicable federal law described in this paragraph, the creditor is not required to provide a separate or duplicative notice pursuant to this section.
(e)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(f)(1)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(A)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1 during the three-year period.
(B)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(C)Substantially modifies an automated decision system and does either of the following:
(i)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period.
(ii)Makes the substantially modified system available to potential deployers.
(2)This subdivision does not apply to a deployer that is subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) with respect to a covered ADS that is used to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
(g)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus five years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(5)Records of any redactions made pursuant to this section.
(h)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to paragraph (15) of subdivision (a) of Section 1798.185 of, and subdivision (b) of Section 1798.199.40 of, the Civil Code.
22756.4.(a)Any of the following public entities The Attorney General may bring a civil action against a developer or deployer who violates this chapter: chapter.
(1)The Attorney General.(2)A district attorney, county counsel of any county within which a city has a population in excess of 750,000, city attorney in a city and county, or city attorney of a city having a population in excess of 750,000.(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.(4)The Civil Rights Department.(5)The Labor Commissioner with respect to employment-related decisions only.(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)(A)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(B)In assessing the amount of the civil penalty pursuant to this paragraph, the court may consider relevant circumstances presented by the parties to the action, including, but not limited to, all of the following:
(i)The nature and severity of the misconduct.
(ii)The number of violations.
(iii)The length of time over which the misconduct occurred and the persistence of the misconduct.
(iv)The willfulness of the misconduct.
(v)The defendants assets, liabilities, and net worth.
(vi)Whether the developer or deployer took reasonable steps to cure the violation during the pendency of the enforcement action.
(c)(1)Before commencing an action to enforce a violation of this chapter, the Attorney General shall issue a notice of violation to the developer or deployer, as appropriate, if a cure is deemed possible by the Attorney General.
(2)If the developer or deployer fails to cure the violation within 60 days after receipt of a notice of violation, the Attorney General may bring an action pursuant to this section.
(3)If the Attorney General finds and can demonstrate that a developer or deployer knowingly violated this chapter, the Attorney General is not required to provide a cure period before commencing an action to seek penalties or relief.
(4)If the Attorney General finds and can demonstrate that a developer has previously been found in violation of this chapter, or has previously been issued a notice of violation of this chapter regardless of whether the developer or deployer cured the noticed violation, the Attorney General is not required to provide a cure period before commencing an action to seek penalties or relief.
(5)This subdivision shall become inoperative January 1, 2030.
22756.5.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do any of the following:
(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(2)Operate aircraft in the national airspace.
(3)Serve a national security or military purpose, if the covered ADS is made available exclusively to a federal government agency for that purpose.
(4)Detect, protect against, or respond to suspected or actual fraudulent electronic payment transactions. fraud or identity theft.
(5)Advertisements.
(6)Detect, protect against, or respond to physical security incidents or illegal activity that threatens the physical safety of natural persons.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
(c)This chapter does not apply to a covered ADS to the extent that its regulation is preempted by federal law.
(d)(1)Except as provided in paragraph (2), this chapter does not apply to a covered entity regulated under the federal Health Insurance Portability and Account Ability Act of 1996 (42 U.S.C. Secs. 1320d to 1320d-9), or a covered entitys business associates for any services rendered to a covered entity, to the extent the covered entity is doing business in the state.
(2)Paragraph (1) does not apply to a consequential decision related to employment or an employment opportunity or a patients eligibility for financial assistance, including discounted care, or to any requirement of subdivision (a) of Section 22756.2.
(e)This chapter does not apply to a medical device subject to oversight by the United States Food and Drug Administration or a pharmaceutical or medical device manufacturers research and development activities that are subject to oversight by the United States Food and Drug Administration, including clinical investigations conducted under Section 312 of Title 21 of the Code of Federal Regulations.
(f)This chapter does not require a person to disclose nonpublic personal information in a manner that would violate the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.).
(g)This chapter does not apply to a medical device or combination product that includes a medical device component subject to oversight by the United States Food and Drug Administration or to a pharmaceutical or medical device manufacturers research and development activities that are subject to oversight by the United States Food and Drug Administration, including clinical investigations conducted under Section 312 of Title 21 of the Code of Federal Regulations, or, in the case of a medical device, investigational device exemptions and other clinical investigations conducted under Part 812 of Title 21 of the Code of Federal Regulations.
(h)This chapter does not apply to an automated decision system used solely to support the development, validation, verification, result interpretation, result classification, or quality assurance or proficiency-testing functions of a laboratory test performed by a laboratory certified under the Clinical Laboratory Improvement Amendments of 1988 (42 U.S.C. Sec. 263a) and regulated under Part 493 of Title 42 of the Code of Federal Regulations to the extent that activity is subject to oversight under that act and those regulations.
(i)This chapter does not apply to an automated decision system used to support manufacturing, quality control, regulatory, safety, pharmacovigilance, quality assurance, batch release, supply chain functions related to the production of a drug, biological product, or medical device, including any GxP uses, to the extent that activity is subject to oversight under any of the following:
(1)Part 210 or 211 of Title 21 of the Code of Federal Regulations.
(2)Parts 600 to 680, inclusive, of Title 21 of the Code of Federal Regulations.
(3)Part 820 of Title 21 of the Code of Federal Regulations, including where that part incorporates by reference quality management system standards for medical devices.
(j)This chapter does not apply to a covered ADS to the extent that the ADS is used to protect against spam and robocall filtering, anti-money laundering and counter-terrorist financing controls, activities relating to technologies used for economic sanctions compliance, and activities relating to technologies used for fraud and scams prevention, including identity verification and consumer identification, if the ADS is not assessing credit risk.
22756.6.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment limited, restricted, or prohibited under any other applicable law.
SEC. 2.Section 51.1.1 is added to the Civil Code, to read:51.1.1.In an action alleging a violation of Section 51 in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated Section 51, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this chapter.
As used in this chapter:
(a)Adverse outcome means either of the following:
(1)A decision that denies, terminates, revokes, or materially reduces or restricts a natural persons access to, eligibility for, selection for, compensation for, or the provision of an opportunity or service.
(2)A decision that results in materially less favorable differentiated price, cost, compensation, or other material terms that are reasonably likely to materially limit, delay, or effectively deny, or otherwise fundamentally alter, a natural persons access to, eligibility for, selection for, compensation for, or the provision of an opportunity or service compared to terms offered to similarly situated natural persons. If a decision outcome imposes materially less favorable differentiated pricing or terms, the decision outcome materially impacts price, cost sharing, compensation, or material terms.
(a)(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(c)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not include a spam email filter, firewall, antivirus software, identity verification or access management tool, calculator, database, dataset, spreadsheet, document validation or authenticity tool, or other compilation of data or information.
(c)(d)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.(C)(B)Accreditation.
(D)(C)Certification.
(E)(D)Admissions and enrollment.
(F)(E)Discipline.
(G)Evaluation.(H)(F)Financial aid and scholarships.
(I)Proctoring.(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:(A)Electricity.(B)Heat.(C)Water.(D)Transportation.(E)Municipal trash and sewage services.(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.(6)(4)Health care and health insurance, including mental health care, dental, and vision.
(7)(5)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.(9)Legal services.(10)Private arbitration.(11)Mediation.(12)Elections as they relate to any of the following:(A)Voting.(B)Redistricting.(C)Voter eligibility and registration.(D)Distribution of voting information.(E)Election administration.(13)(6)Access to government benefits or services or assignment of penalties by a government entity. services.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.(15)(7)Insurance.
(16)Internet and telecommunications access.(d)(e)(1)Covered ADS means an automated decision system that is designed for, or used by, a deployer to profile a natural person in order for the deployer to make or facilitate a consequential decision.
(2)Covered ADS does not mean a GenAI system that is not intended or advertised to make or facilitate a consequential decision.
(e)(f)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(f)(g)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
(g)(h)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision.
(h)(i)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine-tuning of the covered ADS.
(i)(j)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(j)(k)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(k)(l)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(l)(m)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is both of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(2)Express consent does not include an authorization that is any of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(C)Contained within a more general notice, agreement, or set of terms and conditions.
(n)FERPA means the federal Family Educational Rights and Privacy Act of 1974 (20 U.S.C. Sec. 1232g et seq.).
(m)(o)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(n)(p)GenAI system means an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio, that emulates the structure and characteristics of the systems training data.
(o)(q)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(p)(r)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(q)(s)Profile means to use a covered ADS that does both of the following:
(1)Receives, as input data, the personal information of a subject of the consequential decision.
(2)Issues a simplified output, including a score, classification, or recommendation, in order to evaluate the subject or predict outcomes specific to the subject.
(r)(t)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(s)(u)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not include a modification that results from fine-tuning.
(t)(v)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
(u)(w)The definitions of disparate impact and disparate treatment provided in subdivisions (i) and (j) are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the California Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial impact assessment on the covered ADS before January 1, 2027, and annually thereafter.(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct an impact assessment on the covered ADS before initially deploying the covered ADS, or making the covered ADS available to potential deployers, and annually thereafter.(b)In conducting an impact assessment on a covered ADS, a developer shall do all of the following:(1)Describe the purpose of the covered ADS.(2)List and describe all developer-approved uses of the covered ADS.(3)For each developer-approved use, document all of the following:(A)The expected accuracy and reliability of the covered ADS.(B)Whether any disparate treatment is intended to occur and, if so, all of the following:(i)The conditions under which each category of disparate treatment is intended to occur.(ii)Whether each category of disparate treatment is necessary for the developer-approved use.(iii)Whether any alternatives not involving disparate treatment were considered.(C)Whether any disparate impacts are reasonably likely to occur and, if so, all of the following:(i)The conditions under which each category of disparate impact is reasonably likely to occur.(ii)Whether each category of disparate impact is necessary for the developer-approved use.(iii)Whether any alternatives not involving disparate impacts were considered.(4)(A)Beginning January 1, 2030, contract with an independent third-party auditor to assess the developers compliance with this subdivision and complete an audit within six months of the most recent impact assessment and two years after completion of the initial audit.(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.(C)If the deadline for conducting an impact assessment or audit elapses before the impact assessment or audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the impact assessment or audit has been completed.(D)A developer that receives feedback from an auditor pursuant to this paragraph shall do both of the following:(i)Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.(ii)Make a high-level summary of the feedback publicly available at no cost to users of the developers internet website.(c)(a)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(1)A high-level summary of the results of the most recent impact assessment conducted on the covered ADS by the developer pursuant to this chapter.(2)(1)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(3)(2)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(4)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (f) of Section 22756.2.(5)(3)(A)Any technical information necessary for the deployer to comply with this chapter.
(B)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (f) of Section 22756.2.(d)(b)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(1)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(2)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(3)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(e)(c)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all both of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus five years:
(1)The results of any impact assessments conducted on the covered ADS pursuant to this chapter.(2)(1)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.(4)(2)Records of any redactions made pursuant to this chapter.
(f)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent impact assessment conducted on the covered ADS.(g)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.(h)(d)This section does not apply to a developer subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) with respect to a covered ADS that is intended to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
22756.2.(a)(1)Except as provided in paragraph (2), before a deployer finalizes a consequential decision made or facilitated by a covered ADS used by the deployer, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.(C)(B)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(C)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject that will be used to make or facilitate the consequential decision.(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs will be used to make or facilitate the consequential decision.(F)Whether a natural person will review either of the following before the consequential decision is finalized:(i)The outputs of the covered ADS.(ii)The outcome of the consequential decision.(G)(D)The subjects rights under subdivision (c) and the means and timeframe for exercising those rights.
(H)(E)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(3)The disclosure made pursuant to this subdivision may be made through a generalized notice on the deployers internet website or by using an automatic reply mechanism.
(b)(1)After If a deployer finalizes a consequential decision made or facilitated by a covered ADS, ADS that results in an adverse outcome, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within five thirty days:
(A)The personal characteristics or attributes of the subject that the covered ADS measured or assessed to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Whether a natural person reviewed either of the following before the consequential decision was finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)Contact information for the deployer.
(H)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(I)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(J)The subjects rights under subdivision (c) and the means and timeframe for exercising those rights.
(2)Paragraph (1) does not apply to either of the following:
(A)A medical emergency.
(B)The sorting by a covered ADS of resumes based on bona fide occupational qualifications.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(c)After If a deployer finalizes a consequential decision made or facilitated by a covered ADS, ADS that results in an adverse outcome, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall respond to a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision, if feasible.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that determines that rectifying the decision is infeasible shall do both of the following:
(i)Provide the subject with an explanation of the basis for the determination.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(d)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(3)For a consequential decision relating to education, a deployer that is subject to FERPA satisfies the notice and disclosure requirements of this section by providing notice and disclosures through processes and channels that are consistent with FERPA and the deployers FERPA notices and student record access procedures, including, if applicable, notice to a parent or guardian or an eligible student.
(4)A deployer that is subject to FERPA is not required to provide a separate or duplicative notice or disclosure process if the deployer has established a notice or disclosure process to comply with FERPA.
(5)A deployer that is subject to FERPA is not required to establish a separate or duplicative correction and reconsideration process if the deployer has established a correction and reconsideration process to comply with FERPA.
(6)(A)A creditor, with respect to a consequential decision involving the offering, denial, pricing, servicing, or other material terms of credit that provides a notice to a consumer pursuant to the federal Equal Credit Opportunity Act (15 U.S.C. Sec. 1691 et seq.) and, if applicable, the federal Fair Credit Reporting Act (15 U.S.C. Sec. 1681 et seq.), shall be deemed in compliance with the notice or disclosure requirements of Section 22756.2 that relate to the same decision if the notice provided to the consumer pursuant to those laws also satisfies the notice or disclosure requirements of this section.
(B)If a creditor complies with applicable federal law described in this paragraph, the creditor is not required to provide a separate or duplicative notice pursuant to this section.
(e)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(f)(1)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(A)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1 during the three-year period.
(B)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(C)Substantially modifies an automated decision system and does either of the following:
(i)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period.
(ii)Makes the substantially modified system available to potential deployers.
(2)This subdivision does not apply to a deployer that is subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) with respect to a covered ADS that is used to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
(g)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus five years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(5)Any documentation provided to, or received from, auditors pursuant to this chapter.(6)(5)Records of any redactions made pursuant to this section.
(h)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue related to the deployers use of a covered ADS that is raised to that employee.(i)(h)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to paragraph (15) of subdivision (a) of Section 1798.185 of, and subdivision (b) of Section 1798.199.40 of, the Civil Code.
(j)This section shall become operative on January 1, 2027.22756.3.(a)(1)Within 30 days of receiving a request from the Attorney General for an impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the impact assessment to the Attorney General.(2)The Attorney General may share impact assessments with other enforcement entities as necessary for enforcement purposes.(b)(1)The disclosure or sharing of an impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the impact assessment.(2)An impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).(c)Each day a covered ADS is used for which an impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.22756.4.(a)Any of the following public entities may bring a civil action against a developer, deployer, or auditor developer or deployer who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel of any county within which a city has a population in excess of 750,000, city attorney in a city and county, or city attorney of a city having a population in excess of 750,000.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)(A)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(B)In assessing the amount of the civil penalty pursuant to this paragraph, the court may consider relevant circumstances presented by the parties to the action, including, but not limited to, all of the following:
(i)The nature and severity of the misconduct.
(ii)The number of violations.
(iii)The length of time over which the misconduct occurred and the persistence of the misconduct.
(iv)The willfulness of the misconduct.
(v)The defendants assets, liabilities, and net worth.
(c)A developer or deployer who contracts with a third party to perform the developers or deployers duties under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to perform those duties.22756.5.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do any of the following:
(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(2)Operate aircraft in the national airspace.
(3)Serve a national security or military purpose, if the covered ADS is made available exclusively to a federal government agency for that purpose.
(4)Detect, protect against, or respond to suspected or actual fraudulent electronic payment transactions.
(5)Advertisements.
(6)Detect, protect against, or respond to physical security incidents or illegal activity that threatens the physical safety of natural persons.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
(c)This chapter does not apply to a covered ADS to the extent that its regulation is preempted by federal law.
(d)(1)Except as provided in paragraph (2), this chapter does not apply to a covered entity regulated under the federal Health Insurance Portability and Account Ability Act of 1996 (42 U.S.C. Secs. 1320d to 1320d-9), or a covered entitys business associates for any services rendered to a covered entity, to the extent the covered entity is doing business in the state.
(2)Paragraph (1) does not apply to a consequential decision related to employment or an employment opportunity or a patients eligibility for financial assistance, including discounted care, or to any requirement of subdivision (a) of Section 22756.2.
(e)This chapter does not apply to a medical device subject to oversight by the United States Food and Drug Administration or a pharmaceutical or medical device manufacturers research and development activities that are subject to oversight by the United States Food and Drug Administration, including clinical investigations conducted under Section 312 of Title 21 of the Code of Federal Regulations.
(f)This chapter does not require a person to disclose nonpublic personal information in a manner that would violate the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.).
22756.6.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment limited, restricted, or prohibited under any other applicable law.
SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.(e)For purposes of this section:(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.(2)(A)Genetic information means, with respect to any individual, information about any of the following:(i)The individuals genetic tests.(ii)The genetic tests of family members of the individual.(iii)The manifestation of a disease or disorder in family members of the individual.(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.(C)Genetic information does not include information about the sex or age of any individual.(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.(5)Religion includes all aspects of religious belief, observance, and practice.(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:(A)Any combination of those characteristics.(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.(i)In an action alleging a violation of this section in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this section, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.SEC. 2.Section 51.1.1 is added to the Civil Code, to read:51.1.1.In an action alleging a violation of Section 51 in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated Section 51, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean include a spam email filter, firewall, antivirus software, identity and verification or access management tool, calculator, database, dataset, spreadsheet, document validation or authenticity tool, or other compilation of data. data or information.
(c)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions and enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid and scholarships.
(I)Proctoring.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Transportation.
(E)Municipal trash and sewage services.
(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.
(6)Health care and health insurance, including mental health care, dental, and vision.
(7)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
(9)Legal services.
(10)Private arbitration.
(11)Mediation.
(12)Elections as they relate to any of the following:
(A)Voting.
(B)Redistricting.
(C)Voter eligibility and registration.
(D)Distribution of voting information.
(E)Election administration.
(13)Access to government benefits or services or assignment of penalties by a government entity.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.
(15)Insurance.
(16)Internet and telecommunications access.
(d)(1)Covered automated decision system or covered ADS means an automated decision system that is designed or used to for, or used by, a deployer to profile a natural person in order for the deployer to make or facilitate a consequential decision.
(2)Covered ADS does not mean a GenAI system that is not intended or advertised to make or facilitate a consequential decision.
(e)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(f)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
(g)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes. decision.
(h)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine tuning fine-tuning of the covered ADS.
(i)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(j)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(k)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(l)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is all both of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(C)Not contained within a more general notice, agreement, or set of terms and conditions.(2)Express consent does not mean include an authorization that is either any of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(C)Contained within a more general notice, agreement, or set of terms and conditions.
(m)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.(n)GenAI system means an artificial intelligence system that can generate derived synthetic content, including text, images, video, and audio, that emulates the structure and characteristics of the systems training data.
(n)
(o)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(o)
(p)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(q)Profile means to use a covered ADS that does both of the following:
(1)Receives, as input data, the personal information of a subject of the consequential decision.
(2)Issues a simplified output, including a score, classification, or recommendation, in order to evaluate the subject or predict outcomes specific to the subject.
(p)(r)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(q)(s)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not mean include a modification that results from fine tuning. fine-tuning.
(r)(t)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
(s)(u)The definitions of disparate impact and disparate treatment provided in subdivisions (i) and (j) are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the California Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation impact assessment on the covered ADS before January 1, 2027, and annually thereafter.
(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation an impact assessment on the covered ADS before initially deploying the covered ADS, or making the covered ADS available to potential deployers, and annually thereafter.
(b)In conducting a performance evaluation an impact assessment on a covered ADS, a developer shall do all of the following:
(1)Describe the purpose of the covered ADS.
(2)List and describe all developer-approved uses of the covered ADS.
(3)For each developer-approved use, evaluate the expected performance of the covered ADS and document all of the following:
(A)The expected accuracy and reliability of the covered ADS.
(B)Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.(C)(B)Whether any disparate treatment is intended to occur and, if so, all of the following:
(i)The conditions under which each category of disparate treatment is intended to occur.
(ii)Whether each category of disparate treatment is necessary for a the developer-approved use.
(iii)Whether any alternatives not involving disparate treatment were considered.
(D)Any reasonably foreseeable effects of fine tuning on disparate treatment.(E)(C)Whether any disparate impacts are reasonably likely to occur and, if so, all of the following:
(i)The conditions under which each category of disparate impact is reasonably likely to occur.
(ii)Whether each category of disparate impact is necessary for a the developer-approved use.
(iii)Whether any alternatives not involving disparate impacts were considered.
(F)Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.(G)With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.(H)Any reasonably foreseeable effects of fine tuning on disparate impacts.(4)(A)Beginning January 1, 2030, contract with an independent third-party auditor to assess the developers compliance with this subdivision. subdivision and complete an audit within six months of the most recent impact assessment and two years after completion of the initial audit.
(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.
(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)If the deadline for conducting a performance evaluation pursuant to subdivision (a) an impact assessment or audit elapses before the impact assessment or audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the impact assessment or audit has been completed.
(D)A developer that receives feedback from an auditor pursuant to this paragraph shall do both of the following:
(i)Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(ii)Make a high-level summary of the feedback publicly available at no cost to users of the developers internet website.
(c)(1)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(A)The(1)A high-level summary of the results of the most recent performance evaluation impact assessment conducted on the covered ADS by the developer pursuant to this chapter.
(B)(2)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(C)(3)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(D)(4)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (h) (f) of Section 22756.2.
(E)(i)(5)(A)Any technical information necessary for the deployer to comply with this chapter.
(ii)
(B)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (h) (f) of Section 22756.2.
(2)(A)A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets.(B)To the extent that a developer withholds information pursuant to subparagraph (A), the developer shall notify the deployer and provide a basis for the withholding.(d)A developer that receives an impact assessment from an auditor of a deployed covered ADS pursuant to subdivision (b) of Section 22756.3 shall provide all of the following information to any deployer of the covered ADS:(1)Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.(2)Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.(3)Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.(4)An explanation of any steps the deployer can take to mitigate these discrepancies.(e)A developer that receives feedback from an auditor pursuant to this chapter shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.(f)(d)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(1)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(2)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(3)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(g)(e)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus 10 five years:
(1)The results of any performance evaluations impact assessments conducted on the covered ADS pursuant to this chapter.
(2)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.
(4)Records of any redactions made pursuant to this chapter.
(h)(f)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation impact assessment conducted on the covered ADS.
(i)(g)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.
(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
(j)(h)This section does not apply to a developer subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) who uses with respect to a covered ADS that is intended to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
22756.2.(a)(1)Except as provided in paragraph (2), before a deployer finalizes a consequential decision made or facilitated by a covered ADS, ADS used by the deployer, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.
(C)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject that will be used to make or facilitate the consequential decision.
(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.
(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are will be used to make or facilitate the consequential decision.
(F)Whether a natural person will review either of the following before the consequential decision is finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)The subjects rights under subdivisions (b) and (d) subdivision (c) and the means and timeframe for exercising those rights.
(H)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(3)The disclosure made pursuant to this subdivision may be made through a generalized notice on the deployers internet website or by using an automatic reply mechanism.
(b)(1)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within five days:
(A)The personal characteristics or attributes of the subject that the covered ADS used measured or assessed to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Whether a natural person reviewed either of the following before the consequential decision was finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(F)(G)Contact information for the deployer.
(G)(H)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(H)(I)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(I)(J)The subjects rights under subdivision (d) (c) and the means and timeframe for exercising those rights.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency. to either of the following:
(A)A medical emergency.
(B)The sorting by a covered ADS of resumes based on bona fide occupational qualifications.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(c)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall comply with respond to a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision. decision, if feasible.(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that denies a request to correct personal information determines that rectifying the decision is infeasible shall do both of the following:(i)Provide the subject with an explanation of the basis for the denial. determination.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(d)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(e)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(f)(1)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(1)(A)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1 during the three-year period.
(2)(B)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(3)(C)Substantially modifies an automated decision system and does either of the following:
(A)(i)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period.
(B)(ii)Makes the substantially modified system available to potential deployers.
(2)This subdivision does not apply to a deployer that is subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) with respect to a covered ADS that is used to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
(g)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus 10 five years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(5)Any documentation provided to, or received from, auditors pursuant to this chapter.
(6)Records of any redactions made pursuant to this section.
(h)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.
(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue related to the deployers use of a covered ADS that is raised to that employee.
(i)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to paragraph (15) of subdivision (a) of Section 1798.185 of, and subdivision (b) of Section 1798.199.40 of of, the Civil Code.
(j)This section shall become operative on January 1, 2027.
22756.3.(a)(1)Within 30 days of receiving a request from the Attorney General for a performance evaluation an impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the performance evaluation impact assessment to the Attorney General.
(2)The Attorney General may share performance evaluations impact assessments with other enforcement entities as necessary for enforcement purposes.
(b)(1)The disclosure or sharing of a performance evaluation an impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation. impact assessment.
(2)A performance evaluation An impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(c)Each day a covered ADS is used for which a performance evaluation an impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.
22756.4.(a)Any of the following public entities may bring a civil action against a developer, deployer, or auditor who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel of any county within which a city has a population in excess of 750,000, city attorney in a city and county, or city attorney of a city having a population in excess of 750,000.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)(A)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(B)In assessing the amount of the civil penalty pursuant to this paragraph, the court may consider relevant circumstances presented by the parties to the action, including, but not limited to, all of the following:
(i)The nature and severity of the misconduct.
(ii)The number of violations.
(iii)The length of time over which the misconduct occurred and the persistence of the misconduct.
(iv)The willfulness of the misconduct.
(v)The defendants assets, liabilities, and net worth.
(c)A developer or deployer who contracts with a third party to perform the developers or deployers duties under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to perform those duties.
22756.5.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do either any of the following:
(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(2)Operate aircraft in the national airspace.
(3)Serve a national security or military purpose, if the covered ADS is made available exclusively to a federal government agency for that purpose.
(4)Detect, protect against, or respond to suspected or actual fraudulent electronic payment transactions.
(5)Advertisements.
(6)Detect, protect against, or respond to physical security incidents or illegal activity that threatens the physical safety of natural persons.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
(c)This chapter does not apply to a covered ADS to the extent that its regulation is preempted by federal law.
22756.6.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment limited, restricted, or prohibited under any other applicable law.
SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.
(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.
(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.
(e)For purposes of this section:
(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.
(2)(A)Genetic information means, with respect to any individual, information about any of the following:
(i)The individuals genetic tests.
(ii)The genetic tests of family members of the individual.
(iii)The manifestation of a disease or disorder in family members of the individual.
(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.
(C)Genetic information does not include information about the sex or age of any individual.
(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.
(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.
(5)Religion includes all aspects of religious belief, observance, and practice.
(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.
(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:
(A)Any combination of those characteristics.
(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.
(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.
(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.
(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.
(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.
(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.
(i)In an action alleging a violation of this section in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this section, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in performance evaluations and impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(c)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions and enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid and scholarships.
(I)Proctoring.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Transportation.
(E)Municipal trash and sewage services.
(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.
(6)Health care and health insurance, including mental health care, dental, and vision.
(7)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
(9)Legal services.
(10)Private arbitration.
(11)Mediation.
(12)Elections as they relate to any of the following:
(A)Voting.
(B)Redistricting.
(C)Voter eligibility and registration.
(D)Distribution of voting information.
(E)Election administration.
(13)Access to government benefits or services or assignment of penalties by a government entity.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.
(15)Insurance.
(16)Internet and telecommunications access.
(d)Covered automated decision system or covered ADS means an automated decision system that is designed or used to make or facilitate a consequential decision.
(e)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(f)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
(g)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
(h)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine tuning of the covered ADS.
(i)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(j)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(k)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(l)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is all of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(C)Not contained within a more general notice, agreement, or set of terms and conditions.
(2)Express consent does not mean an authorization that is either of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(m)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(n)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(o)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(p)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(q)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not mean a modification that results from fine tuning.
(r)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
(s)The definitions of disparate impact and disparate treatment provided in subdivisions (i) and (j) are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the California Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation on the covered ADS before January 1, 2027, and shall additionally conduct a performance evaluation on the covered ADS under any of the following circumstances: annually thereafter.
(A)Following any substantial modification of the covered ADS by the developer.(B)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.(C)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation on the covered ADS under any of the following circumstances: before initially deploying the covered ADS, or making the covered ADS available to potential deployers, and annually thereafter.
(A)Before initially deploying the covered ADS or making the covered ADS available to potential deployers.(B)Following any substantial modification of the covered ADS by the developer.(C)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.(D)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.(b)In conducting a performance evaluation on a covered ADS, a developer shall do all of the following:
(1)Describe the purpose of the covered ADS.
(2)List and describe all developer-approved uses of the covered ADS.
(3)For each developer-approved use, evaluate the expected performance of the covered ADS and document all of the following:
(A)The expected accuracy and reliability of the covered ADS.
(B)Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.
(C)Whether any disparate treatment is intended to occur and, if so, all of the following:
(i)The conditions under which each disparate treatment is intended to occur.
(ii)Whether each disparate treatment is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate treatment were considered.
(D)Any reasonably foreseeable effects of fine tuning on disparate treatment.
(E)Whether any disparate impacts are reasonably likely to occur and, if so, all of the following:
(i)The conditions under which each disparate impact is reasonably likely to occur.
(ii)Whether each disparate impact is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate impacts were considered.
(F)Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.
(G)With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.
(H)Any reasonably foreseeable effects of fine tuning on disparate impacts.
(4)(A)Contract Beginning January 1, 2030, contract with an independent third-party auditor to assess the developers compliance with this subdivision.
(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.
(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)If the deadline for conducting a performance evaluation pursuant to subdivision (a) elapses before the audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the audit has been completed.
(D)A developer that receives feedback from an auditor pursuant to this paragraph shall do both of the following:
(i)Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(ii)Make a high-level summary of the feedback publicly available at no cost to users of the developers internet website.
(c)(1)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(A)The results of the most recent performance evaluation conducted on the covered ADS by the developer pursuant to this chapter.
(B)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(C)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(D)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(E)(i)Any technical information necessary for the deployer to comply with this chapter.
(ii)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(2)(A)A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a developer withholds information pursuant to subparagraph (A), the developer shall notify the deployer and provide a basis for the withholding.
(d)A developer that receives an impact assessment from an auditor of a deployed covered ADS pursuant to subdivision (b) of Section 22756.3 shall provide all of the following information to any deployer of the covered ADS:
(1)Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(2)Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(3)Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
(4)An explanation of any steps the deployer can take to mitigate these discrepancies.
(e)A developer that receives feedback from an auditor pursuant to this chapter shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(f)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(1)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(2)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(3)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(g)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus 10 years:
(1)The results of any performance evaluations conducted on the covered ADS pursuant to this chapter.
(2)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.
(4)Records of any redactions made pursuant to this chapter.
(h)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
(i)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.
(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
(j)This section does not apply to a developer subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) who uses a covered ADS to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756.
22756.2.(a)(1)Except as provided in paragraph (2), before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.
(C)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject to make or facilitate the consequential decision.
(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.
(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
(F)Whether a natural person will review either of the following before the consequential decision is finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)The subjects rights under subdivisions (b) and (d) and the means and timeframe for exercising those rights.
(H)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(b)(1)Before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS.(2)(A)A deployer may deny a request to opt out of the use of a covered ADS if either of the following is true:(i)The deployer is subject to the federal Gramm-Leach-Bliley Act, and the covered ADS makes or facilitates a consequential decision pursuant to paragraph (7) of subdivision (c) of Section 22756.(ii)The subject of the consequential decision is having a medical emergency.(B)A deployer that denies a request to opt out of the use of a covered ADS pursuant to subparagraph (A) shall provide the subject with an explanation of the basis for the denial.(3)The disclosure made pursuant to this subdivision may be made using an automatic reply mechanism.
(c)(b)(1)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within five days:
(A)The personal characteristics or attributes of the subject that the covered ADS used to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Contact information for the deployer.
(G)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(H)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(I)The subjects rights under subdivision (d) and the means and timeframe for exercising those rights.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(d)(c)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that denies a request to correct personal information shall do both of the following:
(i)Provide the subject with an explanation of the basis for the denial.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(e)(d)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(f)(e)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(g)(1)A deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting more than 6,000 people in a given three-year period shall contract with an independent third-party auditor to conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter.(2)(A)Except pursuant to subparagraph (B), a deployer that contracts with an auditor pursuant to this subdivision shall provide the auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive impact assessment on the covered ADS.(B)A deployer that provides documentation to an auditor pursuant to this paragraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a deployer withholds information, the developer shall notify the auditor and provide a basis for the withholding.(C)This paragraph shall not be construed to require a deployer to collect any personal information from a subject of a consequential decision beyond that which the deployer collects in the ordinary course of business or as necessary to comply with state or federal law.(3)If the deadline for conducting an audit pursuant to paragraph (1) elapses before the audit has been completed, a deployer shall not use the covered ADS to make or facilitate consequential decisions until the audit has been completed.(h)(f)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(1)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1 during the three-year period.
(2)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(3)Substantially modifies an automated decision system and does either of the following:
(A)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period.
(B)Makes the substantially modified system available to potential deployers.
(i)(g)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus 10 years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to opt out of the use of the covered ADS made pursuant to this section.(5)(4)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(6)(5)Any documentation provided to, or received from, auditors pursuant to this chapter.
(7)(6)Records of any redactions made pursuant to this section.
(j)(h)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.
(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue related to the deployers use of a covered ADS that is raised to that employee.
(k)(i)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to subdivision (b) of Section 1798.199.40 of the Civil Code.
(l)(j)This section shall become operative on January 1, 2027.
22756.3.(a)An auditor that conducts an impact assessment on a covered ADS pursuant to subdivision (g) of Section 22756.2 shall do all of the following:(1)(A)Request any information from the deployer of the covered ADS that is reasonably necessary for the auditor to conduct a comprehensive impact assessment.(B)This paragraph applies only to information gathered by the deployer in the ordinary course of business.(2)For each unique developer-approved use of the covered ADS by the deployer, document all of the following:(A)The observed accuracy and reliability of the covered ADS over the relevant period.(B)Whether the observed accuracy and reliability differed materially from the expected accuracy and reliability of the covered ADS, as described in documentation provided by a developer to the deployer pursuant to this chapter.(C)Whether any disparate impacts resulted from the deployers use of the covered ADS and the deployment conditions under which those disparate impacts occurred.(D)Whether each disparate impact was an anticipated disparate impact, as described in documentation provided to the deployer pursuant to this chapter.(3)Whether the deployer used the covered ADS to make or facilitate a consequential decision outside of the scope of a developer-approved use.(4)Whether the deployer assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.(b)(1)After conducting an impact assessment on a covered ADS, an auditor shall do both of the following:(A)Provide the results of the impact assessment to the deployer that contracted with the auditor to perform the impact assessment.(B)Make a high-level summary of the results of the impact assessment publicly available at no cost to users of the auditors internet website.(2)The documentation required by this section shall be both of the following:(A)Provided in English and in any other language that the auditor regularly uses to communicate with deployers.(B)Presented in a manner that ensures the communication clearly and effectively conveys the required information.(3)An auditor shall not make publicly available with the personal information of a subject of a consequential decision made or facilitated by a covered ADS without first obtaining the express consent of the subject.22756.4. 22756.3.(a)(1)Within 30 days of receiving a request from the Attorney General for a performance evaluation or impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the document performance evaluation to the Attorney General.
(2)The Attorney General may share performance evaluations and impact assessments with other enforcement entities as necessary for enforcement purposes.
(b)(1)The disclosure or sharing of a performance evaluation or impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation or impact assessment. evaluation.
(2)A performance evaluation or impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(c)Each day a covered ADS is used for which a performance evaluation or impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.
22756.5. 22756.4.(a)Any of the following public entities may bring a civil action against a developer, deployer, or auditor who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel, counsel of any county within which a city has a population in excess of 750,000, city attorney in a city and county, or city attorney for the jurisdiction in which the violation occurred. of a city having a population in excess of 750,000.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(c)A developer or deployer who contracts with a third party to perform the developers or deployers duties under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to perform those duties.
22756.6.22756.5.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do either of the following:
(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(2)Operate aircraft in the national airspace.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
22756.7.22756.6.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment limited, restricted, or prohibited under any other applicable law.
SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.
(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.
(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.
(e)For purposes of this section:
(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.
(2)(A)Genetic information means, with respect to any individual, information about any of the following:
(i)The individuals genetic tests.
(ii)The genetic tests of family members of the individual.
(iii)The manifestation of a disease or disorder in family members of the individual.
(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.
(C)Genetic information does not include information about the sex or age of any individual.
(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.
(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.
(5)Religion includes all aspects of religious belief, observance, and practice.
(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.
(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:
(A)Any combination of those characteristics.
(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.
(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.
(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.
(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.
(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.
(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.
(i)In an action alleging a violation of this section in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this section, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in performance evaluations and impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(c)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions and enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid and scholarships.
(I)Proctoring.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Transportation.
(E)Municipal trash and sewage services.
(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.
(6)Health care and health insurance, including mental health care, dental, and vision.
(7)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
(9)Legal services.
(10)Private arbitration.
(11)Mediation.
(12)Elections as they relate to any of the following:
(A)Voting.
(B)Redistricting.
(C)Voter eligibility and registration.
(D)Distribution of voting information.
(E)Election administration.
(13)Access to government benefits or services or assignment of penalties by a government entity.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.
(15)Insurance.
(16)Internet and telecommunications access.
(d)Covered automated decision system or covered ADS means an automated decision system that is designed or used to make or facilitate a consequential decision.
(e)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(f)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
(g)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
(h)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine tuning of the covered ADS.
(i)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(j)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(k)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(l)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is all of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(C)Not contained within a more general notice, agreement, or set of terms and conditions.
(2)Express consent does not mean an authorization that is either of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(m)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(n)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(o)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(p)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(q)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not mean a modification that results from fine tuning.
(r)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
(s)The definitions of disparate impact and disparate treatment provided in subdivisions (i) and (j) are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the California Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation on the covered ADS before January 1, 2027, and shall additionally conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Following any substantial modification of the covered ADS by the developer.
(B)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(C)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Before initially deploying the covered ADS or making the covered ADS available to potential deployers.
(B)Following any substantial modification of the covered ADS by the developer.
(C)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(D)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(b)In conducting a performance evaluation on a covered ADS, a developer shall do all of the following:
(1)Describe the purpose of the covered ADS.
(2)List and describe all developer-approved uses of the covered ADS.
(3)For each developer-approved use, evaluate the expected performance of the covered ADS and document all of the following:
(A)The expected accuracy and reliability of the covered ADS.
(B)Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.
(C)Whether any disparate treatment is intended to occur and, if so, all of the following:
(i)The conditions under which each disparate treatment is intended to occur.
(ii)Whether each disparate treatment is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate treatment were considered.
(D)Any reasonably foreseeable effects of fine tuning on disparate treatment.
(E)Whether any disparate impacts are reasonably likely to occur and, if so, all of the following:
(i)The conditions under which each disparate impact is reasonably likely to occur.
(ii)Whether each disparate impact is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate impacts were considered.
(F)Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.
(G)With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.
(H)Any reasonably foreseeable effects of fine tuning on disparate impacts.
(4)(A)Contract with an independent third-party auditor to assess the developers compliance with this subdivision.
(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.
(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)If the deadline for conducting a performance evaluation pursuant to subdivision (a) elapses before the audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the audit has been completed.
(D)A developer that receives feedback from an auditor pursuant to this paragraph shall do both of the following:
(i)Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(ii)Make a high-level summary of the feedback publicly available at no cost to users of the developers internet website.
(c)(1)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(A)The results of the most recent performance evaluation conducted on the covered ADS by the developer pursuant to this chapter.
(B)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(C)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(D)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(E)(i)Any technical information necessary for the deployer to comply with this chapter.
(ii)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(2)(A)A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a developer withholds information pursuant to subparagraph (A), the developer shall notify the deployer and provide a basis for the withholding.
(d)A developer that receives an impact assessment from an auditor of a deployed covered ADS pursuant to subdivision (b) of Section 22756.3 shall provide all of the following information to any deployer of the covered ADS:
(1)Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(2)Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(3)Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
(4)An explanation of any steps the deployer can take to mitigate these discrepancies.
(e)A developer that receives feedback from an auditor pursuant to this chapter shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(f)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(1)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(2)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(3)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(g)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus 10 years:
(1)The results of any performance evaluations conducted on the covered ADS pursuant to this chapter.
(2)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.
(4)Records of any redactions made pursuant to this chapter.
(h)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
(i)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.
(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
22756.2.(a)(1)Except as provided in paragraph (2), before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.
(C)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject to make or facilitate the consequential decision.
(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.
(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
(F)Whether a natural person will review either of the following before the consequential decision is finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)The subjects rights under subdivisions (b) and (d) and the means and timeframe for exercising those rights.
(H)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(b)(1)Before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS.
(2)(A)A deployer may deny a request to opt out of the use of a covered ADS if either of the following is true:
(i)The deployer is subject to the federal Gramm-Leach-Bliley Act, and the covered ADS makes or facilitates a consequential decision pursuant to paragraph (7) of subdivision (c) of Section 22756.
(ii)The subject of the consequential decision is having a medical emergency.
(B)A deployer that denies a request to opt out of the use of a covered ADS pursuant to subparagraph (A) shall provide the subject with an explanation of the basis for the denial.
(c)(1)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within five days:
(A)The personal characteristics or attributes of the subject that the covered ADS used to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Contact information for the deployer.
(G)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(H)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(I)The subjects rights under subdivision (d) and the means and timeframe for exercising those rights.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(d)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that denies a request to correct personal information shall do both of the following:
(i)Provide the subject with an explanation of the basis for the denial.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(e)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(f)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(g)(1)A deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting more than 5,999 6,000 people in a given three-year period shall contract with an independent third-party auditor to conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter.
(2)(A)Except pursuant to subparagraph (B), a deployer that contracts with an auditor pursuant to this subdivision shall provide the auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive impact assessment on the covered ADS.
(B)A deployer that provides documentation to an auditor pursuant to this paragraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a deployer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)This paragraph shall not be construed to require a deployer to collect any personal information from a subject of a consequential decision beyond that which the deployer collects in the ordinary course of business or as necessary to comply with state or federal law.
(3)If the deadline for conducting an audit pursuant to paragraph (1) elapses before the audit has been completed, a deployer shall not use the covered ADS to make or facilitate consequential decisions until the audit has been completed.
(h)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(1)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5,999 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1 during the three-year period.
(2)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5,999 6,000 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(3)Substantially modifies an automated decision system and does either of the following:
(A)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 5,999 6,000 people in a given three-year period.
(B)Makes the substantially modified system available to potential deployers.
(i)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus 10 years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to opt out of the use of the covered ADS made pursuant to this section.
(5)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(6)Any documentation provided to, or received from, auditors pursuant to this chapter.
(7)Records of any redactions made pursuant to this section.
(j)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.
(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue related to the deployers use of a covered ADS that is raised to that employee.
(k)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to subdivision (b) of Section 1798.199.40 of the Civil Code.
(l)This section shall become operative on January 1, 2027.
22756.3.(a)An auditor that conducts an impact assessment on a covered ADS pursuant to subdivision (g) of Section 22756.2 shall do all of the following:
(1)(A)Request any information from the deployer of the covered ADS that is reasonably necessary for the auditor to conduct a comprehensive impact assessment.
(B)This paragraph applies only to information gathered by the deployer in the ordinary course of business.
(2)For each unique developer-approved use of the covered ADS by the deployer, document all of the following:
(A)The observed accuracy and reliability of the covered ADS over the relevant period.
(B)Whether the observed accuracy and reliability differed materially from the expected accuracy and reliability of the covered ADS, as described in documentation provided by a developer to the deployer pursuant to this chapter.
(C)Whether any disparate impacts resulted from the deployers use of the covered ADS and the deployment conditions under which those disparate impacts occurred.
(D)Whether each disparate impact was an anticipated disparate impact, as described in documentation provided to the deployer pursuant to this chapter.
(3)Whether the deployer used the covered ADS to make or facilitate a consequential decision outside of the scope of a developer-approved use.
(4)Whether the deployer assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(b)(1)After conducting an impact assessment on a covered ADS, an auditor shall do both of the following:
(A)Provide the results of the impact assessment to the deployer that contracted with the auditor to perform the impact assessment.
(B)Make a high-level summary of the results of the impact assessment publicly available at no cost to users of the auditors internet website.
(2)The documentation required by this section shall be both of the following:
(A)Provided in English and in any other language that the auditor regularly uses to communicate with deployers.
(B)Presented in a manner that ensures the communication clearly and effectively conveys the required information.
(3)An auditor shall not make publicly available with the personal information of a subject of a consequential decision made or facilitated by a covered ADS without first obtaining the express consent of the subject.
22756.4.(a)(1)Within 30 days of receiving a request from the Attorney General for a performance evaluation or impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the document to the Attorney General.
(2)The Attorney General may share performance evaluations and impact assessments with other enforcement entities as necessary for enforcement purposes.
(b)(1)The disclosure or sharing of a performance evaluation or impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation or impact assessment.
(2)A performance evaluation or impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(c)Each day a covered ADS is used for which a performance evaluation or impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.
22756.5.(a)Any of the following public entities may bring a civil action against a developer, deployer, or auditor who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(c)A developer or deployer who contracts with a third party to perform the developers or deployers duties under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to perform those duties.
22756.6.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do either of the following:
(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(2)Operate aircraft in the national airspace.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
22756.7.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment limited, restricted, or prohibited under any other applicable law.
SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.
(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.
(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.
(e)For purposes of this section:
(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.
(2)(A)Genetic information means, with respect to any individual, information about any of the following:
(i)The individuals genetic tests.
(ii)The genetic tests of family members of the individual.
(iii)The manifestation of a disease or disorder in family members of the individual.
(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.
(C)Genetic information does not include information about the sex or age of any individual.
(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.
(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.
(5)Religion includes all aspects of religious belief, observance, and practice.
(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.
(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:
(A)Any combination of those characteristics.
(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.
(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.
(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.
(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.
(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.
(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.
(i)In an action alleging a violation of this section in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this section, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in performance evaluations and impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(c)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions and enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid and scholarships.
(I)Proctoring.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Transportation.
(E)Municipal trash and sewage services.
(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.
(6)Health care and health insurance, including mental health care, dental, and vision.
(7)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
(9)Legal services.
(10)Private arbitration.
(11)Mediation.
(12)Elections as they relate to any of the following:
(A)Voting.
(B)Redistricting.
(C)Voter eligibility and registration.
(D)Distribution of voting information.
(E)Election administration.
(13)Access to government benefits or services or assignment of penalties by a government entity.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.
(15)Insurance.
(16)Internet and telecommunications access.
(d)Covered automated decision system or covered ADS means an automated decision system that is designed or used to make or facilitate a consequential decision.
(e)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(f)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
(g)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
(h)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine tuning of the covered ADS.
(i)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(j)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(k)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(l)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is all of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(C)Not contained within a more general notice, agreement, or set of terms and conditions.
(2)Express consent does not mean an authorization that is either of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(m)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(n)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(o)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(p)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(q)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not mean a modification that results from fine tuning.
(r)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
(s)The definitions of disparate impact and disparate treatment provided in subdivisions (i) and (j) are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the California Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation on the covered ADS before January 1, 2027, and shall additionally conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Following any substantial modification of the covered ADS by the developer.
(B)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(C)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Before initially deploying the covered ADS or making the covered ADS available to potential deployers.
(B)Following any substantial modification of the covered ADS by the developer.
(C)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(D)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(b)In conducting a performance evaluation on a covered ADS, a developer shall do all of the following:
(1)Describe the purpose of the covered ADS.
(2)List and describe all developer-approved uses of the covered ADS.
(3)For each developer-approved use, evaluate the expected performance of the covered ADS and document all of the following:
(A)The expected accuracy and reliability of the covered ADS.
(B)Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.
(C)Whether any disparate treatment is intended to occur and, if so, all of the following:
(i)The conditions under which each disparate treatment is intended to occur.
(ii)Whether each disparate treatment is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate treatment were considered.
(D)Any reasonably foreseeable effects of fine tuning on disparate treatment.
(E)Whether any disparate impacts are reasonably likely to occur and, if so, all of the following:
(i)The conditions under which each disparate impact is reasonably likely to occur.
(ii)Whether each disparate impact is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate impacts were considered.
(F)Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.
(G)With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.
(H)Any reasonably foreseeable effects of fine tuning on disparate impacts.
(4)(A)Contract with an independent third-party auditor to assess the developers compliance with this subdivision.
(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.
(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)If the deadline for conducting a performance evaluation pursuant to subdivision (a) elapses before the audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the audit has been completed.
(I)
(D)A developer that receives feedback from an auditor pursuant to this paragraph shall do both of the following:
(i)Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(ii)Make a high-level summary of the feedback publicly available at no cost to users of the developers internet website.
(c)(1)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(A)The results of the most recent performance evaluation conducted on the covered ADS by the developer pursuant to this chapter.
(B)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(C)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(D)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(E)(i)Any technical information necessary for the deployer to comply with this chapter.
(ii)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(2)(A)A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a developer withholds information pursuant to subparagraph (A), the developer shall notify the deployer and provide a basis for the withholding.
(d)A developer that receives an impact assessment from an auditor of a deployed covered ADS pursuant to subdivision (b) of Section 22756.3 shall provide all of the following information to any deployer of the covered ADS:
(A)
(1)Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(B)
(2)Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(C)
(3)Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
(D)
(4)An explanation of any steps the deployer can take to mitigate these discrepancies.
(e)A developer that receives feedback from an auditor pursuant to this chapter shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(f)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(A)
(1)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(B)
(2)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(C)
(3)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(g)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus 10 years:
(1)The results of any performance evaluations conducted on the covered ADS pursuant to this chapter.
(2)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.
(4)Records of any redactions made pursuant to this chapter.
(h)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
(i)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.
(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
22756.2.(a)(1)Except as provided in paragraph (2), before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.
(C)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject to make or facilitate the consequential decision.
(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.
(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
(F)Whether a natural person will review either of the following before the consequential decision is finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)The subjects rights under subdivisions (b) and (d) and the means and timeframe for exercising those rights.
(H)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(b)(1)Before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS.
(2)(A)A deployer may deny a request to opt out of the use of a covered ADS if either of the following is true:
(i)The deployer is subject to the federal Gramm-Leach-Bliley Act, and the covered ADS makes or facilitates a consequential decision pursuant to paragraph (7) of subdivision (c) of Section 22756.
(ii)The subject of the consequential decision is having a medical emergency.
(B)A deployer that denies a request to opt out of the use of a covered ADS pursuant to subparagraph (A) shall provide the subject with an explanation of the basis for the denial.
(c)(1)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within five days:
(A)The personal characteristics or attributes of the subject that the covered ADS used to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Contact information for the deployer.
(G)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(H)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(I)The subjects rights under subdivision (d) and the means and timeframe for exercising those rights.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(d)After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that denies a request to correct personal information shall do both of the following:
(i)Provide the subject with an explanation of the basis for the denial.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(e)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(f)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(g)(1)A deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting more than 5,999 people in a given three-year period shall contract with an independent third-party auditor to conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter.
(2)(A)Except pursuant to subparagraph (B), a deployer that contracts with an auditor pursuant to this subdivision shall provide the auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive impact assessment on the covered ADS.
(B)A deployer that provides documentation to an auditor pursuant to this paragraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a deployer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)This paragraph shall not be construed to require a deployer to collect any personal information from a subject of a consequential decision beyond that which the deployer collects in the ordinary course of business or as necessary to comply with state or federal law.
(3)If the deadline for conducting an audit pursuant to paragraph (1) elapses before the audit has been completed, a deployer shall not use the covered ADS to make or facilitate consequential decisions until the audit has been completed.
(h)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(1)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5,999 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1 during the three-year period.
(2)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5,999 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(3)Substantially modifies an automated decision system and does either of the following:
(A)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 5999 5,999 people in a given three-year period.
(B)Makes the substantially modified system available to potential deployers.
(i)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus 10 years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to opt out of the use of the covered ADS made pursuant to this section.
(5)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(6)Any documentation provided to, or received from, auditors pursuant to this chapter.
(7)Records of any redactions made pursuant to this section.
(j)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.
(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue related to the deployers use of a covered ADS that is raised to that employee.
(k)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to subdivision (b) of Section 1798.199.40 of the Civil Code.
(l)This section shall become operative on January 1, 2027.
22756.3.(a)An auditor that conducts an impact assessment on a covered ADS pursuant to subdivision (g) of Section 22756.2 shall do all of the following:
(1)(A)Request any information from the deployer of the covered ADS that is reasonably necessary for the auditor to conduct a comprehensive impact assessment.
(B)This paragraph applies only to information gathered by the deployer in the ordinary course of business.
(2)For each unique developer-approved use of the covered ADS by the deployer, document all of the following:
(A)The observed accuracy and reliability of the covered ADS over the relevant period.
(B)Whether the observed accuracy and reliability differed materially from the expected accuracy and reliability of the covered ADS, as described in documentation provided by a developer to the deployer pursuant to this chapter.
(C)Whether any disparate impacts resulted from the deployers use of the covered ADS and the deployment conditions under which those disparate impacts occurred.
(D)Whether each disparate impact was an anticipated disparate impact, as described in documentation provided to the deployer pursuant to this chapter.
(3)Whether the deployer used the covered ADS to make or facilitate a consequential decision outside of the scope of a developer-approved use.
(4)Whether the deployer assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(b)(1)After conducting an impact assessment on a covered ADS, an auditor shall do both of the following:
(A)Provide the results of the impact assessment to the deployer that contracted with the auditor to perform the impact assessment.
(B)Make a high-level summary of the results of the impact assessment publicly available at no cost to users of the auditors internet website.
(2)The documentation required by this section shall be both of the following:
(A)Provided in English and in any other language that the auditor regularly uses to communicate with deployers.
(B)Presented in a manner that ensures the communication clearly and effectively conveys the required information.
(3)An auditor shall not make publicly available with the personal information of a subject of a consequential decision made or facilitated by a covered ADS without first obtaining the express consent of the subject.
22756.4.(a)(1)Within 30 days of receiving a request from the Attorney General for a performance evaluation or impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the document to the Attorney General.
(2)The Attorney General may share performance evaluations and impact assessments with other enforcement entities as necessary for enforcement purposes.
(b)(1)The disclosure or sharing of a performance evaluation or impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation or impact assessment.
(2)A performance evaluation or impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(c)Each day a covered ADS is used for which a performance evaluation or impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.
22756.5.(a)Any of the following public entities may bring a civil action against a developer, deployer, or auditor who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(c)A developer or deployer who contracts with a third party to perform the developers or deployers duties under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to perform those duties.
22756.6.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do either of the following:
(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(2)Operate aircraft in the national airspace.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
22756.7.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment limited, restricted, or prohibited under any other applicable law.
SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.
(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.
(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.
(e)For purposes of this section:
(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.
(2)(A)Genetic information means, with respect to any individual, information about any of the following:
(i)The individuals genetic tests.
(ii)The genetic tests of family members of the individual.
(iii)The manifestation of a disease or disorder in family members of the individual.
(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.
(C)Genetic information does not include information about the sex or age of any individual.
(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.
(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.
(5)Religion includes all aspects of religious belief, observance, and practice.
(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.
(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:
(A)Any combination of those characteristics.
(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.
(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.
(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.
(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.
(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.
(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.
(i)In an action alleging a violation of this section in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 25 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this section. section, and a failure to comply with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code shall not, by itself, give rise to a presumption of unlawful intent.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in performance evaluations and impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(c)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions and enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid and scholarships.
(I)Proctoring.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet and telecommunications access.(E)(D)Transportation.
(F)(E)Municipal trash and sewage services.
(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.
(6)Health care and health insurance, including mental health care, dental, and vision.
(7)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
(9)Legal services.
(10)Private arbitration.
(11)Mediation.
(12)Elections as they relate to any of the following:
(A)Voting.
(B)Redistricting.
(C)Voter eligibility and registration.
(D)Distribution of voting information.
(E)Election administration.
(13)Access to government benefits or services or assignment of penalties by a government entity.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.
(15)Insurance.
(16)Internet and telecommunications access.
(d)Covered automated decision system or covered ADS means an automated decision system that makes or facilitates is designed or used to make or facilitate a consequential decision.
(e)Credit score means a credit score, as defined in Section 1785.15.1 of the Civil Code, from a consumer credit reporting agency, as defined in Section 1785.3 of the Civil Code.
(e)
(f)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or through by contracting with a third party. party for that purpose.
(f)
(g)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or through by contracting with a third party. party for those purposes.
(g)
(h)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine tuning of the covered ADS.
(h)
(i)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(i)
(j)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(j)
(k)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(k)
(l)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is all of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(C)Not contained within a more general notice, agreement, or set of terms and conditions.
(2)Express consent does not mean an authorization that is either of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(l)
(m)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(m)
(n)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(n)
(o)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(o)
(p)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(p)
(q)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not mean a modification that results from fine tuning.
(q)
(r)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation on the covered ADS before January 1, 2027, and shall additionally conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Following any substantial modification of the covered ADS by the developer.
(B)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(C)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Before initially deploying the covered ADS or making the covered ADS available to potential deployers.
(B)Following any substantial modification of the covered ADS by the developer.
(C)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(D)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(b)In conducting a performance evaluation on a covered ADS, a developer shall do all of the following:
(1)Describe the purpose of the covered ADS.
(2)List and describe all developer-approved uses of the covered ADS.
(3)For each developer-approved use, assess evaluate the expected performance of the covered ADS and document all of the following:
(A)The expected accuracy and reliability of the covered ADS.
(B)Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.
(4)For each developer-approved use, assess whether(C)Whether any disparate treatment is intended to occur and document and, if so, all of the following:
(A)Whether the covered ADS is intended to treat individuals or groups of individuals differently on the basis of a protected characteristic.(B)For each disparate treatment identified under subparagraph (A), describe all of the following:(i)The conditions under which the each disparate treatment is intended to occur.
(ii)Whether the each disparate treatment is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate treatment were considered.
(C)(D)Any reasonably foreseeable effects of fine tuning on disparate treatment.
(5)For each developer-approved use, assess whether(E)Whether any disparate impacts are reasonably likely to occur and document and, if so, all of the following:
(A)Whether the covered ADS is reasonably likely to treat groups of individuals who share a protected characteristic differently.(B)For each disparate impact identified under subparagraph (A), describe all of the following:(i)The conditions under which that each disparate impact is reasonably likely to occur.
(ii)Whether the each disparate impact is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate impacts were considered.
(C)(F)Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.
(D)(G)With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.
(E)(H)Any reasonably foreseeable effects of fine tuning on disparate impacts.
(6)(4)(A)Contract with an independent third-party auditor to assess the developers compliance with this subdivision.
(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.
(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)If the deadline for conducting a performance evaluation pursuant to subdivision (a) elapses before the audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the audit has been completed.
(I)A developer that receives feedback from an auditor pursuant to this paragraph shall do both of the following:
(i)Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(ii)Make a high-level summary of the feedback publicly available at no cost to users of the developers internet website.
(c)(1)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(A)The results of the most recent performance evaluation conducted on the covered ADS by the developer pursuant to this chapter.
(B)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(C)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(D)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(E)(i)Any technical information necessary for the deployer to comply with this chapter.
(ii)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(2)(A)A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a developer withholds information pursuant to subparagraph (A), the developer shall notify the deployer and provide a basis for the withholding.
(d)A developer that receives an impact assessment from an auditor of a deployed covered ADS pursuant to subdivision (b) of Section 22756.3 shall provide all of the following information to any deployer of the covered ADS:
(A)Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(B)Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(C)Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
(D)An explanation of any steps the deployer can take to mitigate these discrepancies.
(e)A developer that receives feedback from an auditor pursuant to this chapter shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(f)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(A)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(B)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(C)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(g)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the covered ADS remains deployed or developer deploys the covered ADS or makes the covered ADS available to potential deployers plus 10 years:
(1)The results of any performance evaluations conducted on the covered ADS pursuant to this chapter.
(2)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.
(4)Records of any redactions made pursuant to this chapter.
(h)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
(i)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.
(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
22756.2.(a)(1)Except as provided in paragraph (2), before finalizing a deployer finalizes a consequential decision made or facilitated by a covered ADS, a the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.
(C)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject to make or facilitate the consequential decision.
(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.
(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
(F)Whether a natural person will review either of the following before the consequential decision is finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)The subjects rights under subdivisions (b) and (c) (d) and the means and timeframe for exercising those rights.
(H)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(b)(1)Before finalizing a deployer finalizes a consequential decision made or facilitated by a covered ADS, a the deployer shall provide any subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS.
(2)(A)A deployer may deny a request to opt out of the use of a covered ADS if either of the following is true:
(i)The deployer is subject to the federal Gramm-Leach-Bliley Act, and the covered ADS makes or facilitates a consequential decision pursuant to paragraph (7) of subdivision (c) of Section 22756.
(ii)The subject of the consequential decision is having a medical emergency.
(B)A deployer that denies a request to opt out of the use of a covered ADS pursuant to subparagraph (A) shall provide the subject with an explanation of the basis for the denial.
(c)(1)After a deployer finalizes a consequential decision is finalized, made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure shall be provided containing all of the following information within five days, and the disclosure shall include all of the information: days:
(A)The personal characteristics or attributes of the subject that the covered ADS measured or assessed used to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Contact information for the deployer.
(G)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(H)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(I)The subjects rights under subdivision (d) and the means and timeframe for exercising those rights.
(2)Paragraph (1) is does not applicable apply if the subject of the consequential decision is having a medical emergency.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(d)After finalizing a deployer finalizes a consequential decision made or facilitated by a covered ADS, a the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that denies a request to correct personal information shall do both of the following:
(i)Provide the subject with an explanation of the basis for the denial.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request. request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that the original decision is was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision is was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(e)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(f)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(g)(1)A deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting more than 5999 5,999 people in a given three-year period shall contract with an independent third-party auditor to conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter.
(2)(A)Except pursuant to subparagraph (B), a deployer that contracts with an auditor pursuant to this subdivision shall provide the auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive impact assessment on the covered ADS.
(B)A deployer that provides documentation to an auditor pursuant to this paragraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a deployer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)This paragraph shall not be construed to require a deployer to collect any personal information from a subject of a consequential decision beyond that which the deployer collects in the ordinary course of business or as necessary to comply with state or federal law.
(3)If the deadline for conducting an audit pursuant to paragraph (1) elapses before the audit has been completed, a deployer shall not use the covered ADS to make or facilitate consequential decisions until the audit has been completed.
(h)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(1)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5999 5,999 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1. 22756.1 during the three-year period.
(2)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5999 5,999 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(3)Substantially modifies an automated decision system and does either of the following:
(A)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 5999 people in a given three-year period.
(B)Makes the substantially modified system available to potential deployers.
(i)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS remains deployed plus 10 years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to opt out of the use of the covered ADS made pursuant to this section.
(5)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(6)Any documentation provided to, or received from, auditors pursuant to this chapter.
(7)Records of any redactions made pursuant to this section.
(j)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.
(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue related to the deployers use of a covered ADS that is raised to that employee.
(k)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to subdivision (b) of Section 1798.199.40 of the Civil Code.
(l)This section shall become operative on January 1, 2027.
22756.3.(a)An auditor that conducts an impact assessment on a covered ADS pursuant to subdivision (g) of Section 22756.2 shall do all of the following:
(1)(A)Request any information from the deployer of the covered ADS that is reasonably necessary for the auditor to conduct a comprehensive impact assessment.
(B)This paragraph applies only to information gathered by the deployer in the ordinary course of business.
(2)Document all developer-approved uses of the covered ADS that the deployer utilized during the relevant period.(3)(2)For each unique developer-approved use, use of the covered ADS by the deployer, document all of the following:
(A)The observed accuracy and reliability of the covered ADS over the relevant period.
(B)Whether the observed accuracy and reliability differed materially from the expected accuracy and reliability of the covered ADS, as described in documentation provided by a developer to the deployer pursuant to this chapter.
(C)Whether any disparate impacts resulted from the deployers use of the covered ADS and the deployment conditions under which those disparate impacts occurred.
(D)Whether each disparate impact was an anticipated disparate impact, as described in documentation provided to the deployer pursuant to this chapter.
(4)(3)Whether the deployer used the covered ADS to make or facilitate a consequential decision outside of the scope of a developer-approved use.
(5)(4)Whether the deployer assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(b)(1)After conducting an impact assessment on a covered ADS, an auditor shall provide the results of the impact assessment to do both of the following:
(A)The Provide the results of the impact assessment to the deployer that contracted with the auditor to perform the impact assessment.
(B)The developer of the covered ADS.(B)Make a high-level summary of the results of the impact assessment publicly available at no cost to users of the auditors internet website.
(2)The documentation required by this section shall be both of the following:
(A)Provided in English and in any other language that the auditor regularly uses to communicate with developers and deployers.
(B)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the developer and deployer. information.
(3)An auditor shall not provide a developer make publicly available with the personal information of a subject of a consequential decision made or facilitated by a covered ADS without first obtaining the express consent of the subject.
22756.4.(a)(1)Within 30 days of receiving a request from the Attorney General for a performance evaluation or impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the document to the Attorney General.
(2)The Attorney General may share performance evaluations and impact assessments with other enforcement entities as necessary for enforcement purposes.
(b)(1)The disclosure or sharing of a performance evaluation or impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation or impact assessment.
(2)A performance evaluation or impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(c)Each day a covered ADS is used for which a performance evaluation or impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.
22756.5.(a)Any of the following public entities may bring a civil action against a developer or deployer developer, deployer, or auditor who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
(c)A developer or deployer who contracts with a third party to perform the developers or deployers duties under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to perform those duties.
22756.6.(a)This chapter does not apply to an automated decision system the sole purpose of which is to do either of the following:
(a)(1)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(b)(2)Operate aircraft in the national airspace.
(b)The use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
22756.7.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment that are limited, restricted, or prohibited under any other applicable law.
22756.8.A developer or deployer who contracts with a third party to comply with duties required under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to comply with this chapter.SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.
(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.
(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.
(e)For purposes of this section:
(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.
(2)(A)Genetic information means, with respect to any individual, information about any of the following:
(i)The individuals genetic tests.
(ii)The genetic tests of family members of the individual.
(iii)The manifestation of a disease or disorder in family members of the individual.
(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.
(C)Genetic information does not include information about the sex or age of any individual.
(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.
(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.
(5)Religion includes all aspects of religious belief, observance, and practice.
(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.
(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:
(A)Any combination of those characteristics.
(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.
(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.
(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.
(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.
(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.
(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.
(i)In an action alleging a violation of this section in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have committed caused or facilitated the violation, the extent to which the defendant complied with Chapter 25 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether a the defendant violated this section.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which the defendants development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have committed caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether a the defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in performance evaluations and impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
As used in this chapter:
(a)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Automated decision system means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(c)Consequential decision means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
(1)Employment-related decisions.
(2)Education and vocational training as they relate to any of the following:
(A)Assessment and placement.
(B)Detecting student cheating and plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions and enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid and scholarships.
(I)Proctoring.
(3)Housing and lodging as they relate to any of the following:
(A)Rental or short-term housing and lodging.
(B)Home appraisals.
(C)Rental subsidies.
(D)Publicly supported housing.
(4)Any of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet and telecommunications access.
(E)Transportation.
(F)Municipal trash and sewage services.
(5)Family planning, adoption services, reproductive services, and assessments related to child protective services.
(6)Health care and health insurance, including mental health care, dental, and vision.
(7)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(8)The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
(9)Legal services.
(10)Private arbitration.
(11)Mediation.
(12)Elections as they relate to any of the following:
(A)Voting.
(B)Redistricting.
(C)Voter eligibility and registration.
(D)Distribution of voting information.
(E)Election administration.
(13)Access to government benefits or services or assignment of penalties by a government entity.
(14)Places of public accommodation, as defined in Section 55.52 of the Civil Code.
(d)Covered automated decision system or covered ADS means an automated decision system that makes or facilitates a consequential decision.
(e)Deployer means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or through a third party.
(f)Developer means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or through a third party.
(g)(1)Developer-approved use means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision.
(2)Developer-approved use includes any reasonably foreseeable fine tuning of the covered ADS.
(h)Disparate impact means a differential effect on a group of individuals who share a protected characteristic.
(i)Disparate treatment means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
(j)Employment-related decision means a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
(k)(1)Express consent means an affirmative written authorization that is granted in response to a notice that is all of the following:
(A)Clear, meaningful, and prominent.
(B)Conveyed in a manner that a natural person would notice and understand it.
(C)Not contained within a more general notice, agreement, or set of terms and conditions.
(2)Express consent does not mean an authorization that is either of the following:
(A)Inferred from inaction.
(B)Obtained through the use of a dark pattern, as defined in Section 56.18 of the Civil Code.
(l)Fine-tune means to adjust the model parameters of an automated decision system through exposure to additional data.
(m)Labor Commissioner means Chief of the Division of Labor Standards Enforcement.
(n)Personal information has the same meaning as defined in Section 1798.140 of the Civil Code.
(o)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(p)(1)Substantial modification means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
(2)Substantial modification does not mean a modification that results from fine tuning.
(q)Trade secret has the same meaning as in Section 3426.1 of the Civil Code.
22756.1.(a)(1)With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation on the covered ADS before January 1, 2027, and shall additionally conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Following any substantial modification of the covered ADS by the developer.
(B)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(C)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(2)With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation on the covered ADS under any of the following circumstances:
(A)Before initially deploying the covered ADS or making the covered ADS available to potential deployers.
(B)Following any substantial modification of the covered ADS by the developer.
(C)Following any fine tuning of the covered ADS by the developer that materially changes the uses or outputs of the covered ADS.
(D)No more than one year after the developer last conducted a performance evaluation on the covered ADS, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
(b)In conducting a performance evaluation on a covered ADS, a developer shall do all of the following:
(1)Describe the purpose of the covered ADS.
(2)List and describe all developer-approved uses of the covered ADS.
(3)For each developer-approved use, assess the expected performance of the covered ADS and document all of the following:
(A)The expected accuracy and reliability of the covered ADS.
(B)Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.
(4)For each developer-approved use, assess whether any disparate treatment is intended to occur and document all of the following:
(A)Whether the covered ADS is intended to treat individuals or groups of individuals differently on the basis of a protected characteristic.
(B)For each disparate treatment identified under subparagraph (A), describe all of the following:
(i)The conditions under which the disparate treatment is intended to occur.
(ii)Whether the disparate treatment is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate treatment were considered.
(C)Any reasonably foreseeable effects of fine tuning on disparate treatment.
(5)For each developer-approved use, assess whether any disparate impacts are reasonably likely to occur and document all of the following:
(A)Whether the covered ADS is reasonably likely to treat groups of individuals who share a protected characteristic differently.
(B)For each disparate impact identified under subparagraph (A), describe all of the following:
(i)The conditions under which that disparate impact is reasonably likely to occur.
(ii)Whether the disparate impact is necessary for a developer-approved use.
(iii)Whether any alternatives not involving disparate impacts were considered.
(C)Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.
(D)With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.
(E)Any reasonably foreseeable effects of fine tuning on disparate impacts.
(6)(A)Contract with an independent third-party auditor to assess the developers compliance with this subdivision.
(B)(i)Except pursuant to clause (ii), a developer that contracts with an auditor pursuant to this paragraph shall provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance.
(ii)A developer that provides documentation to an auditor pursuant to this subparagraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)If the deadline for conducting a performance evaluation pursuant to subdivision (a) elapses before the audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the audit has been completed.
(c)(1)A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
(A)The results of the most recent performance evaluation conducted on the covered ADS by the developer pursuant to this chapter.
(B)For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
(C)For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
(D)An explanation of the deployers responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(E)(i)Any technical information necessary for the deployer to comply with this chapter.
(ii)A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(2)(A)A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a developer withholds information pursuant to subparagraph (A), the developer shall notify the deployer and provide a basis for the withholding.
(d)A developer that receives an impact assessment from an auditor of a deployed covered ADS pursuant to subdivision (b) of Section 22756.3 shall provide all of the following information to any deployer of the covered ADS:
(A)Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(B)Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
(C)Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
(D)An explanation of any steps the deployer can take to mitigate these discrepancies.
(e)A developer that receives feedback from an auditor pursuant to this chapter shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
(f)A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
(A)Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
(B)Provided in English and in any other language the developer regularly uses to communicate with deployers.
(C)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
(g)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the covered ADS remains deployed or available to potential deployers plus 10 years:
(1)The results of any performance evaluations conducted on the covered ADS pursuant to this chapter.
(2)Any documentation provided to deployers pursuant to this chapter.
(3)Any documentation provided to, or received from, auditors pursuant to this chapter.
(4)Records of any redactions made pursuant to this chapter.
(h)It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
(i)(1)A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developers compliance with this chapter.
(2)A developer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
22756.2.(a)(1)Except as provided in paragraph (2), before finalizing a consequential decision made or facilitated by a covered ADS, a deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
(A)A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
(B)The name, version number, and developer of the covered ADS.
(C)Whether the deployers use of the covered ADS is within the scope of a developer-approved use and a description of that use.
(D)(i)The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
(ii)The sources of personal information collected from the subject to make or facilitate the consequential decision.
(iii)Any key parameters known to disproportionately affect the outcome of the consequential decision.
(E)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
(F)Whether a natural person will review either of the following before the consequential decision is finalized:
(i)The outputs of the covered ADS.
(ii)The outcome of the consequential decision.
(G)The subjects rights under subdivisions (b) and (c) and the means and timeframe for exercising those rights.
(H)(i)Contact information for the deployer.
(ii)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(iii)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(2)Paragraph (1) does not apply if the subject of the consequential decision is having a medical emergency.
(b)(1)Before finalizing a consequential decision made or facilitated by a covered ADS, a deployer shall provide any subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS.
(2)(A)A deployer may deny a request to opt out of the use of a covered ADS if either of the following is true:
(i)The deployer is subject to the federal Gramm-Leach-Bliley Act, and the covered ADS makes or facilitates a consequential decision pursuant to paragraph (7) of subdivision (c) of Section 22756.
(ii)The subject of the consequential decision is having a medical emergency.
(B)A deployer that denies a request to opt out of the use of a covered ADS pursuant to subparagraph (A) shall provide the subject with an explanation of the basis for the denial.
(c)(1)After a consequential decision is finalized, a plain language written disclosure shall be provided within five days, and the disclosure shall include all of the information:
(A)The personal characteristics or attributes of the subject that the covered ADS measured or assessed to make or facilitate the consequential decision.
(B)The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
(C)Any key parameters that disproportionately affected the outcome of the consequential decision.
(D)The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
(E)The role that the ADS played in making the consequential decision and whether any human judgment was involved.
(F)Contact information for the deployer.
(G)Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
(H)Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
(I)The subjects rights under subdivision (d) and the means and timeframe for exercising those rights.
(2)Paragraph (1) is not applicable if the subject of the consequential decision is having a medical emergency.
(3)A deployer that provides documentation to a subject of a consequential decision pursuant to this subdivision shall ensure the documentation is all of the following:
(A)Transmitted directly to the subject.
(B)Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
(C)Made available in formats that are accessible to people who are blind or have other disabilities.
(D)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
(d)After finalizing a consequential decision made or facilitated by a covered ADS, a deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
(1)(A)Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision.
(B)A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request.
(C)(i)If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
(D)A deployer that denies a request to correct personal information shall do both of the following:
(i)Provide the subject with an explanation of the basis for the denial.
(ii)Provide the subject with a reasonable opportunity to request that the deployer delete the subjects personal information.
(2)(A)Appeal the outcome of the consequential decision.
(B)A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request.
(C)(i)If a deployer determines that the original decision is incorrect, the deployer shall, within 30 days of making the determination, rectify the decision.
(ii)If a deployer determines that the original decision is correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
(D)A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
(e)(1)(A)A deployer that provides documentation to the subject of a consequential decision pursuant to this section may make reasonable redactions for the purpose of protecting trade secrets.
(B)To the extent that a deployer withholds information pursuant to paragraph (1), the deployer shall notify the subject and provide a basis for the withholding.
(2)A deployer that is required by another state or federal law to provide substantially similar notice to a subject of a consequential decision need not duplicatively provide notice to the subject under this section.
(f)A deployers collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
(g)(1)A deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting more than 5999 people in a given three-year period shall contract with an independent third-party auditor to conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter.
(2)(A)Except pursuant to subparagraph (B), a deployer that contracts with an auditor pursuant to this subdivision shall provide the auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive impact assessment on the covered ADS.
(B)A deployer that provides documentation to an auditor pursuant to this paragraph may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a deployer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
(C)This paragraph shall not be construed to require a deployer to collect any personal information from a subject of a consequential decision beyond that which the deployer collects in the ordinary course of business or as necessary to comply with state or federal law.
(3)If the deadline for conducting an audit pursuant to paragraph (1) elapses before the audit has been completed, a deployer shall not use the covered ADS to make or facilitate consequential decisions until the audit has been completed.
(h)A deployer that does any of the following assumes the responsibilities of a developer under this chapter:
(1)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5999 people in a given three-year period, if the deployer did not receive documentation from the developer of the covered ADS pursuant to subdivision (c) of Section 22756.1.
(2)Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 5999 people in a given three-year period, if the deployers use of the covered ADS is outside the scope of a developer-approved use.
(3)Substantially modifies an automated decision system and does either of the following:
(A)Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 5999 people in a given three-year period.
(B)Makes the substantially modified system available to potential deployers.
(i)A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the covered ADS remains deployed plus 10 years:
(1)Any documentation received from developers pursuant to this chapter.
(2)Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
(3)Any requests to correct personal information made pursuant to this section.
(4)Any requests to opt out of the use of the covered ADS made pursuant to this section.
(5)Any requests to appeal the outcome of a consequential decision made pursuant to this section.
(6)Any documentation provided to, or received from, auditors pursuant to this chapter.
(7)Records of any redactions made pursuant to this section.
(j)(1)A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployers compliance with this chapter.
(2)A deployer shall require an employee designated pursuant to this subdivision to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
(k)In addition to the requirements of this section, a deployer that is a business subject to the California Consumer Privacy Act of 2018 (Title 1.81.5 (commencing with Section 1798.100)) is subject to any privacy-related opt-out and access regulation duly adopted by the California Privacy Protection Agency pursuant to subdivision (b) of Section 1798.199.40 of the Civil Code.
(l)This section shall become operative on January 1, 2027.
22756.3.(a)An auditor that conducts an impact assessment on a covered ADS pursuant to subdivision (g) of Section 22756.2 shall do all of the following:
(1)Request any information from the deployer of the covered ADS that is reasonably necessary for the auditor to conduct a comprehensive impact assessment.
(2)Document all developer-approved uses of the covered ADS that the deployer utilized during the relevant period.
(3)For each unique developer-approved use, document all of the following:
(A)The observed accuracy and reliability of the covered ADS over the relevant period.
(B)Whether the observed accuracy and reliability differed materially from the expected accuracy and reliability of the covered ADS, as described in documentation provided by a developer to the deployer pursuant to this chapter.
(C)Whether any disparate impacts resulted from the deployers use of the covered ADS and the deployment conditions under which those disparate impacts occurred.
(D)Whether each disparate impact was an anticipated disparate impact, as described in documentation provided to the deployer pursuant to this chapter.
(4)Whether the deployer used the covered ADS to make or facilitate a consequential decision outside of the scope of a developer-approved use.
(5)Whether the deployer assumed the responsibilities of a developer pursuant to subdivision (h) of Section 22756.2.
(b)(1)After conducting an impact assessment on a covered ADS, an auditor shall provide the results of the impact assessment to both of the following:
(A)The deployer that contracted with the auditor to perform the impact assessment.
(B)The developer of the covered ADS.
(2)The documentation required by this section shall be both of the following:
(A)Provided in English and in any other language that the auditor regularly uses to communicate with developers and deployers.
(B)Presented in a manner that ensures the communication clearly and effectively conveys the required information to the developer and deployer.
(3)An auditor shall not provide a developer with the personal information of a subject of a consequential decision made or facilitated by a covered ADS without first obtaining the express consent of the subject.
22756.4.(a)(1)Within 30 days of receiving a request from the Attorney General for a performance evaluation or impact assessment prepared pursuant to this chapter, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the document to the Attorney General.
(2)The Attorney General may share performance evaluations and impact assessments with other enforcement entities as necessary for enforcement purposes.
(b)(1)The disclosure or sharing of a performance evaluation or impact assessment pursuant to subdivision (a) does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation or impact assessment.
(2)A performance evaluation or impact assessment disclosed or shared pursuant to subdivision (a) is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(c)Each day a covered ADS is used for which a performance evaluation or impact assessment has not been submitted to the Attorney General pursuant to this section is an additional violation of this section.
22756.5.(a)Any of the following public entities may bring a civil action against a developer or deployer who violates this chapter:
(1)The Attorney General.
(2)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(3)A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
(4)The Civil Rights Department.
(5)The Labor Commissioner with respect to employment-related decisions only.
(b)A court may award a prevailing plaintiff who brings an action pursuant to subdivision (a) all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)A civil penalty of up to twenty-five thousand dollars ($25,000) per violation.
22756.6.This chapter does not apply to an automated decision system the sole purpose of which is to do either of the following:
(a)Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems.
(b)Operate aircraft in the national airspace.
22756.7.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that is limited, restricted, or prohibited under any other applicable law.
(d)This chapter does not authorize disparate impacts or disparate treatment that are limited, restricted, or prohibited under any other applicable law.
22756.8.A developer or deployer who contracts with a third party to comply with duties required under this chapter, other than those duties related to auditing, is subject to liability under this chapter for the third partys failure to comply with this chapter.
SEC. 2.Section 51 of the Civil Code, as amended by Section 2.5 of Chapter 779 of the Statutes of 2024, is amended to read:51.(a)This section shall be known, and may be cited, as the Unruh Civil Rights Act.
(b)All persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
(c)This section shall not be construed to confer any right or privilege on a person that is conditioned or limited by law or that is applicable alike to persons of every sex, color, race, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, citizenship, primary language, or immigration status, or to persons regardless of their genetic information.
(d)Nothing in this section shall be construed to require any construction, alteration, repair, structural or otherwise, or modification of any sort whatsoever, beyond that construction, alteration, repair, or modification that is otherwise required by other provisions of law, to any new or existing establishment, facility, building, improvement, or any other structure, nor shall anything in this section be construed to augment, restrict, or alter in any way the authority of the State Architect to require construction, alteration, repair, or modifications that the State Architect otherwise possesses pursuant to other laws.
(e)For purposes of this section:
(1)Disability means any mental or physical disability as defined in Sections 12926 and 12926.1 of the Government Code.
(2)(A)Genetic information means, with respect to any individual, information about any of the following:
(i)The individuals genetic tests.
(ii)The genetic tests of family members of the individual.
(iii)The manifestation of a disease or disorder in family members of the individual.
(B)Genetic information includes any request for, or receipt of, genetic services, or participation in clinical research that includes genetic services, by an individual or any family member of the individual.
(C)Genetic information does not include information about the sex or age of any individual.
(3)Medical condition has the same meaning as defined in subdivision (i) of Section 12926 of the Government Code.
(4)Race is inclusive of traits associated with race, including, but not limited to, hair texture and protective hairstyles. Protective hairstyles includes, but is not limited to, such hairstyles as braids, locs, and twists.
(5)Religion includes all aspects of religious belief, observance, and practice.
(6)Sex includes, but is not limited to, pregnancy, childbirth, or medical conditions related to pregnancy or childbirth. Sex also includes, but is not limited to, a persons gender. Gender means sex, and includes a persons gender identity and gender expression. Gender expression means a persons gender-related appearance and behavior whether or not stereotypically associated with the persons assigned sex at birth.
(7)Sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status includes any of the following:
(A)Any combination of those characteristics.
(B)A perception that the person has any particular characteristic or characteristics within the listed categories or any combination of those characteristics.
(C)A perception that the person is associated with a person who has, or is perceived to have, any particular characteristic or characteristics, or any combination of characteristics, within the listed categories.
(8)Sexual orientation has the same meaning as defined in subdivision (s) of Section 12926 of the Government Code.
(f)A violation of the right of any individual under the federal Americans with Disabilities Act of 1990 (Public Law 101-336) shall also constitute a violation of this section.
(g)Verification of immigration status and any discrimination based upon verified immigration status, where required by federal law, shall not constitute a violation of this section.
(h)Nothing in this section shall be construed to require the provision of services or documents in a language other than English, beyond that which is otherwise required by other provisions of federal, state, or local law, including Section 1632.
(i)In an action alleging a violation of this section in which an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have committed or facilitated the violation, the extent to which the defendant complied with Chapter 25 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether a defendant violated this section.
SEC. 3.Article 3 (commencing with Section 12959) is added to Chapter 6 of Part 2.8 of Division 3 of Title 2 of the Government Code, to read:3.Automated Decision Systems12959.In an action alleging a violation of this chapter in which an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have committed or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether a defendant violated this chapter.
SEC. 4.The Legislature finds and declares that Section 1 of this act, which adds Chapter 24.6 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in performance evaluations and impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
|
Date
|
Status
|
|---|---|
| Aug 30, 2026 | Read second time. Ordered to third reading. |
| Aug 28, 2026 | Read third time and amended. Ordered to second reading. |
| Aug 24, 2026 | Read second time. Ordered to third reading. |
| Aug 21, 2026 | Read second time and amended. Ordered returned to second reading. |
| Aug 21, 2026 | Ordered to second reading. |
| Aug 21, 2026 | From inactive file. |
| Sep 13, 2025 | Ordered to inactive file at the request of Senator Wiener. |
| Sep 8, 2025 | Read second time. Ordered to third reading. |
| Sep 5, 2025 | Read third time and amended. Ordered to second reading. |
| Aug 29, 2025 | Read second time. Ordered to third reading. |
| Aug 29, 2025 | From committee: Do pass. (Ayes 5. Noes 2.) (August 29). |
| Aug 18, 2025 | In committee: Referred to suspense file. |
| Jul 17, 2025 | Read second time and amended. Re-referred to Com. on APPR. |
| Jul 16, 2025 | From committee: Amend, and do pass as amended and re-refer to Com. on APPR. (Ayes 11. Noes 2.) (July 15). |
| Jul 3, 2025 | From committee chair, with author's amendments: Amend, and re-refer to committee. Read second time, amended, and re-referred to Com. on JUD. |
| Jun 26, 2025 | In committee: Hearing postponed by committee. |
| Jun 11, 2025 | Referred to Com. on JUD. |
| Jun 3, 2025 | In Senate. Read first time. To Com. on RLS. for assignment. |
| Jun 2, 2025 | Read third time. Passed. Ordered to the Senate. (Ayes 50. Noes 16. Page 1931.) |
| May 27, 2025 | Read second time. Ordered to third reading. |
| May 23, 2025 | From committee: Do pass. (Ayes 10. Noes 3.) (May 23). |
| May 14, 2025 | In committee: Set, first hearing. Referred to APPR. suspense file. |
| May 5, 2025 | Re-referred to Com. on APPR. |
| May 1, 2025 | Read second time and amended. |
| Apr 30, 2025 | From committee: Amend, and do pass as amended and re-refer to Com. on APPR. (Ayes 8. Noes 3.) (April 29). |
| Apr 23, 2025 | From committee: Do pass and re-refer to Com. on JUD. (Ayes 9. Noes 3.) (April 22). Re-referred to Com. on JUD. |
| Apr 21, 2025 | Re-referred to Com. on P. & C.P. |
| Apr 10, 2025 | From committee chair, with author's amendments: Amend, and re-refer to Com. on P. & C.P. Read second time and amended. |
| Mar 28, 2025 | In committee: Set, first hearing. Hearing canceled at the request of author. |
| Mar 10, 2025 | Referred to Coms. on P. & C.P. and JUD. |
| Feb 21, 2025 | From printer. May be heard in committee March 23. |
| Feb 20, 2025 | Read first time. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Aug 29, 2025 | Do pass | Senate Standing Committee on Appropriations |
Passed |
|
| Aug 18, 2025 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Jul 15, 2025 | Do pass as amended, but first amend, and re-refer to the Committee on [Appropriations] | Senate Standing Committee on Judiciary |
Passed |
|
| Jun 2, 2025 | AB 1018 Bauer-Kahan Assembly Third Reading | Assembly Floor |
Passed |
|
| May 23, 2025 | Do pass. | Assembly Standing Committee on Appropriations |
Passed |
|
| Apr 29, 2025 | Do pass as amended and be re-referred to the Committee on [Appropriations] | Assembly Standing Committee on Judiciary |
Passed |
|
| Apr 22, 2025 | Do pass and be re-referred to the Committee on [Judiciary] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| Abundance Institute | ** Against | |
| ACCE Action (Alliance of Californians for Community Empowerment) | ** Support | |
| Advanced Medical Technology Association | ** Against | |
| Aerospace and Defense Alliance of California | ** Against | |
| AMERICAN COLLEGE OF OBSTETRICIANS & GYNECOLOGISTS - DISTRICT IX | ** Against | |
| American Council Of Life Insurers | ** Against | |
| American Federation of Musicians, Local 7 | ** Support | |
| American Financial Services Association | ** Against | |
| American Innovators Network | ** Against | |
| American Property Casualty Insurance Association | ** Against | |
| American Staffing Association | ** Against | |
| America’s Physician Groups | ** Against | |
| America'S Physician Groups | ** Against | |
| Asian Americans Advancing Justice Southern California | ** Support | |
| Associated General Contractors Of California | ** Against | |
| Association Of California Life And Health Insurance Companies | ** Against | |
| Association of Independent California Colleges and Universities | ** Against | |
| Association of National Advertisers | ** Against | |
| Business Software Alliance | ** Against | |
| Business Software Association | ** Against | |
| CalBroadband | ** Against | |
| Calcom Association | ** Against | |
| California Association Of Clerks & Election Officials | ** Against | |
| California Association Of Health Plans | ** Against | |
| California Bankers Association | ** Against | |
| California Center for Movement Legal Services | ** Support | |
| California Chamber Of Commerce | ** Against | |
| California Civil Liberties Advocacy | ** Support | |
| California Community Banking Network | ** Against | |
| California Community Foundation | ** Support | |
| California Credit Union League | ** Against | |
| California Dental Association | ** Against | |
| California Department of Insurance | ** Against | |
| California Employment Lawyers Association | ** Support | |
| California Federation of Labor Unions, Afl-cio | ** Support | |
| California Financial Services Association | ** Against | |
| California Fuels And Convenience Alliance | ** Against | |
| California Hospital Association | ** Against | |
| California Immigrant Policy Center | ** Support | |
| California Initiative for Technology and Democracy | ** Support | |
| California Initiative on Technology and Democracy | ** Support | |
| California Land Title Association | ** Against | |
| California Life Sciences | ** Against | |
| California Manufacturers And Technology Association | ** Against | |
| California Medical Association | ** Against | |
| California Mortgage Bankers Association | ** Against | |
| California National Organization For Women | ** Support | |
| California Nurses Association | ** Support | |
| California Professional Firefighters | ** Support | |
| California Radiological Society | ** Against | |
| California Retailers Association | ** Against | |
| California School Employees Association | ** Support | |
| California Staffing Professionals | ** Against | |
| California Women's Law Center | ** Support | |
| CFT – a Union of Educators & Classified Professionals, AFT, AFL-CIO | ** Support | |
| Chamber Of Progress | ** Against | |
| Child Care Law Center | ** Support | |
| Citizen's Privacy Coalition | ** Support | |
| Citizen’s Privacy Coalition | ** Support | |
| Civil Justice Association Of California | ** Against | |
| College Board | ** Against | |
| Common Sense Media | ** Support | |
| Computer And Communications Industry Association | ** Against | |
| Connected Commerce Council | ** Against | |
| Consumer Attorneys Of California | ** Support | |
| Consumer Data Industry Association | ** Against | |
| Consumer Federation of America | ** Support | |
| Consumer Federation Of California | ** Support | |
| Consumer Reports | ** Support | |
| Consumer Technology Association | ** Against | |
| Contra Costa County | ** Against | |
| County of Ventura | ** Against | |
| County Welfare Directors Association of California | ** Against | |
| Courage California | ** Support | |
| CTIA | ** Against | |
| Delta Dental Of California | ** Against | |
| East Bay Community Law Center | ** Support | |
| Economic Security California Action | ** Support | |
| El Dorado County Chamber of Commerce | ** Against | |
| El Dorado Hills Chamber of Commerce | ** Against | |
| Electronic Privacy Information Center | ** Support | |
| Elk Grove Chamber Of Commerce | ** Against | |
| End Child Poverty California Powered by Grace | ** Support | |
| Equal Rights Advocates | ** Support | |
| Fair Housing Advocates of Northern California | ** Support | |
| Family Business Association of California | ** Against | |
| Felony Murder Elimination Project | ** Support | |
| Financial Technology Association | ** Against | |
| Fiserv | ** Against | |
| Folsom Chamber Of Commerce | ** Against | |
| Greenlining Institute | ** Support | |
| hollywood chamber of commerce | ** Against | |
| Housing Now! | ** Support | |
| Independent Insurance Agents & Brokers of California, Inc. | ** Against | |
| Innovative Lending Platform Association | ** Against | |
| Internet Works | ** Against | |
| Internet.Works | ** Against | |
| Judicial Council of California | ** Against | |
| Justice2jobs Coalition | ** Support | |
| Kaiser Permanente | ** Against | |
| Kapor Center | ** Support | |
| LAANE (Los Angeles Alliance for a New Economy) | ** Support | |
| LA Defensa | ** Support | |
| Lawyers Committee for Civil Rights of the San Francisco Bay Area | ** Support | |
| lincoln chamber of commerce | ** Against | |
| Los Angeles County | ** Against | |
| Los Angeles County Business Federation | ** Against | |
| Mortgage Bankers Association | ** Against | |
| Mujeres Unidas Y Activas | ** Support | |
| National Association of Mutual Insurance Companies | ** Against | |
| Northern California District Council of the International Longshore and Warehouse Union | ** Support | |
| Oakland Privacy | ** Support | |
| OCHIN, INC. | ** Against | |
| Orange County Business Council | ** Against | |
| Orange County Employees Association | ** Support | |
| Other | ** Against | |
| Pacific Association Of Domestic Insurance Companies | ** Against | |
| Parent Voices | ** Support | |
| Personal Insurance Federation Of California | ** Against | |
| Powerswitch Action | ** Support | |
| Privacy Rights Clearinghouse | ** Support | |
| Psinapse Techology | ** Against | |
| Public Citizen | ** Support | |
| Public Risk Innovation, Solutions, and Management | ** Against | |
| PUBLIC RISK INNOVATION, SOLUTIONS, AND MANAGEMENT (PRISM) | ** Against | |
| Rancho Cordova Chamber of Commerce | ** Against | |
| Receivables Management Association International | ** Against | |
| Restoring Hope California | ** Support | |
| Rocklin Area Chamber of Commerce | ** Against | |
| Rockson? (not in list) | ** Against | |
| Roseville Area Chamber Of Commerce | ** Against | |
| Rubicon Programs | ** Support | |
| Santa Maria Valley Chamber of Commerce | ** Against | |
| Securities Industry And Financial Markets Association | ** Against | |
| Security Industry Association | ** Against | |
| SEIU California | ** Support | |
| Shingle Springs/cameron Park Chamber of Commerce | ** Against | |
| Silicon Valley Leadership Group | ** Against | |
| Smart Justice California | ** Support | |
| Society For Human Resource Management | ** Against | |
| Software & Information Industry Association | ** Against | |
| Software Information Industry Association | ** Against | |
| STAR Staffing | ** Against | |
| Surveillance Resistance Lab | ** Support | |
| Sutter Health | ** Against | |
| Techca | ** Against | |
| TechEquity Action (source) | ** Support | |
| Technet | ** Against | |
| Tech Oversight Project | ** Support | |
| The Payments Coalition | ** Against | |
| Transparency Coalition.ai | ** Support | |
| UDW/AFSCME Local 3930 | ** Support | |
| UFCW - Western States Council | ** Support | |
| UNITED CHAMBER ADVOCACY NETWORK UCAN | ** Against | |
| University of California | ** Against | |
| VALLEY INDUSTRY AND COMMERCE ASSOCIATION | ** Against | |
| Vision y Compromiso | ** Support | |
| Women'S Foundation California | ** Support | |
| Working Partnerships USA | ** Support | |
| Worksafe | ** Support | |
| Yuba Sutter Chamber of Commerce | ** Against |
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-4171 | AB 1018 |
THIRD READING
Bill No: AB 1018
Author: Bauer-Kahan (D), et al.
Amended: 8/28/26 in Senate
Vote: 21
SENATE JUDICIARY COMMITTEE: 11-2, 7/15/25
AYES: Umberg, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Wahab, Weber Pierson, Wiener
NOES: Niello, Valladares
SENATE APPROPRIATIONS COMMITTEE: 5-2, 8/29/25
AYES: Caballero, Cabaldon, Grayson, Richardson, Wahab
NOES: Seyarto, Dahle
ASSEMBLY FLOOR: 50-16, 6/2/25 - See last page for vote
SUBJECT: Automated decision systems
SOURCE: TechEquity Action
DIGEST:
This bill regulates the use of “automated decision systems” (ADS). It places obligations on developers and deployers of such systems designed or used to help make “consequential decisions.”
Senate Floor amendments of 8/28/26 change various definitions, remove state agencies from the scope of the bill, alter the transparency requirements, exempt smaller employers from some provisions, narrow enforcement authority to only the Attorney General, insert a right to cure, add a number of exemptions, and make clarifying and technical changes.
Senate Floor amendments of 8/21/26 add various exemptions and significantly narrow the bill by, among other things, removing requirements to conduct risk assessments and audits, and limiting certain rights to “adverse outcomes.”
Senate Floor amendments of 9/5/25 narrow the scope of this bill by, in part, carving out certain generative artificial intelligence (AI) systems and adding several exemptions. The amendments also reduce the burden on developers and deployers by, in part, reducing the document retention timeline and including factors for courts to consider in assessing civil penalties.
ANALYSIS:
Existing law:
- Establishes the California Consumer Privacy Act (CCPA), which grants consumers certain rights with regard to their personal information, including enhanced notice, access, and disclosure; the right to deletion; the right to restrict the sale of information; and protection from discrimination for exercising these rights. It places attendant obligations on businesses to respect those rights. (Civil (Civ.) Code § 1798.100 et seq.)
- Establishes the Consumer Privacy Rights Act (CPRA), which amends the CCPA and creates the California Privacy Protection Agency (PPA), which is charged with implementing these laws, promulgating regulations, and carrying out enforcement actions. (Civ. Code § 1798.100 et seq.; Proposition 24 (2020).)
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Government (Gov.) Code § 12930.)
- Establishes the Fair Employment and Housing Act (FEHA). (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that meets specified criteria. (Civ. Code § 3426.1(d).)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms. (Gov. Code § 11546.45.5.)
This bill:
- Requires a developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer to provide the deployer with specified information.
- Dictates the process and form that any documentation required to be provided to a deployer by a developer must take.
- Requires a deployer, before it finalizes a consequential decision made or facilitated by a covered ADS, and after it finalizes such a decision that results in an adverse outcome, to provide any subject of that decision with specified disclosures and the ability to request further information. A deployer that employs more than 25 employees must also provide the subject with an opportunity to correct any incorrect personal information used and to appeal the adverse consequential decision according to specified processes. Dictates the process and form that documentation required to be provided to a subject by a deployer must take.
- Provides that a deployer’s collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
- Provides the conditions under which a deployer assumes the responsibilities of a developer.
- Imposes retention requirements on developers and deployers with respect to specified documentation.
- Allows for the redaction of certain information revealing trade secrets, as provided, but requires disclosure of such withholding and the basis for it.
- Authorizes enforcement actions against developers or deployers to be brought solely by the Attorney General. It further provides a limited right to cure until January 1, 2030.
- Authorizes a court to award a prevailing plaintiff in such actions specified remedies, including a civil penalty of up to $25,000 per violation. It provides courts with relevant circumstances to be considered in assessing the civil penalty.
- Provides certain exemptions from specified requirements, including for deployers subject to the federal Family Educational Rights and Privacy Act, the federal Equal Credit Opportunity Act, and the federal Fair Credit Reporting Act; entities regulated by HIPAA; a developer subject to the federal Gramm-Leach-Bliley Act; medical devices; laboratory testing; ADS used to support development of drugs, biological products, and medical devices; and ADS to the extent they are used to protect against things such as spam or money laundering.
- Clarifies the relevance of compliance, or the failure to comply, with these provisions in actions pursuant to the Unruh Civil Rights Act and FEHA.
- Defines AI, ADS, trade secret, adverse outcome, and personal information in accordance with existing law and defines other relevant terms.
Background
ADS, especially those powered by AI, are being increasingly deployed in a multitude of contexts, including employment, housing, education, and health care. Major transparency and fairness concerns have been raised about the use of ADS to make consequential decisions, essentially determinations with significant legal or other material effect on people’s lives.
This bill seeks to regulate the development and deployment, by both public and private actors, of “covered ADS,” essentially ADS designed for or used by a deployer to help make “consequential decisions,” defined broadly to include decisions, determinations, or any other actions made about a person that relates to the provision of, access to, or the eligibility, selection, or compensation for, certain specified goods, services, or decisions, as provided. Developers are required to provide deployers certain information, and deployers are required to provide certain notices to subjects of consequential decisions and afford certain rights to them.
This bill is sponsored by TechEquity Action. It is supported by a variety of labor, consumer, privacy, and other advocacy groups, including Common Sense Media and Consumer Reports. It is opposed by a number of industry associations, including the Consumer Technology Association and the American Staffing Association. For a more thorough discussion, please see the Senate Judiciary Committee analysis of this bill, which is incorporated by reference herein.
Comments
According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- Unknown, potentially significant costs (local funds, General Fund) to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the hundreds of millions of dollars annually statewide. For example, counties will be required to review all of their systems for compliance—including tools used for online testing, hiring, permitting, case management, risk assessments, and property management. They will also incur costs of audits, training, staffing, disclosures, appeals process, data retention, sending out notices, and potential legal exposure. If developers raise their cost as a result of this bill, these costs will be passed onto the counties. Counties may claim reimbursement of those costs if the Commission on State Mandates determines that this bill creates a new program or imposes a higher level of service on local agencies.
- The State Controller’s Office (SCO) reports that it, and all other state governmental entities would be considered a “deployer,” of an ADS, as defined in the bill. If SCO implemented one or multiple ADS systems that meet the criteria for oversight in AB 1018, SCO would anticipate needing the following General Fund Resources:
- 2025-26: $1,408,000 for 7.0 two-year limited-term positions and one-time costs;
- 2026-27: $1,384,000 for 7.0 two-year limited-term positions; and,
- 2027-28 and ongoing: $761,000 for 4.0 permanent positions.
SCO indicates that these estimates reflect the requirements for a large, complex agency such as the SCO managing sensitive personal and financial data across automated systems. Actual staffing may be adjusted depending on the complexity and scale of the deployed ADS. Some functions may overlap or be handled by existing staff with appropriate upskilling, while external contractors could augment capacity for short-term surges, such as during initial implementation or major audits. If AB 1018 becomes law, the SCO would need 5–7 information security staff—including a dedicated supervisor—for the initial implementation of AB 1018, with 3–4 (including the supervisor) required for ongoing security and compliance operations related to automated decision systems.
- Costs (special fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce Labor Code violations, possibly in the millions of dollars, annually. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- The Department of Justice (DOJ) estimates a fiscal impact of $2.5 million or less (General Fund, Unfair Competition Law Fund). The DOJ notes that implementation of this bill will be dependent upon the appropriation of funds. The DOJ will be unable to absorb the costs to comply with or implement the requirements of the bill within existing budgeted resources. The Consumer Protection Section (CPS), Civil Rights Enforcement Section (CRES) and Healthcare Rights & Access Section (HRA) within the Public Rights Division of the DOJ anticipate increased workloads in enforcing this bill beginning on January 1, 2026, and ongoing. The workload includes investigating and prosecuting violations. The Sections will require the following additional resources:
- CPS: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution;
- CRES: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution; and,
- HRA: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution.
- The Civil Rights Department (CRD) estimates a fiscal impact over $2 million (General Fund). CRD notes that this bill places requirements on ADS developers and deployers. It also provides the CRD with the authority to bring a civil action to enforce compliance with this bill. CRD anticipates requesting eight positions, including researchers/ research data specialists, associate governmental program analysts, and attorneys, approximately $2.2 million for FY 26-27, and $2.1 million for FY 27-28 and ongoing. In addition to covering the costs of the new positions, these resources would also fund an ongoing contract with an expert consultant, IT and other administrative costs, and a one-time job analysis that would be required for one of the research positions. These resources would be necessary to support the complex legal and technological analysis required to enforce the components of this bill.
- The State Water Board estimates a significant fiscal impact (General Fund, special funds) likely in the millions of dollars per year. The State Water Board reports, as written, this bill’s definition of automated decision system is vague, ambiguous, and could encompass many current tools used, like excel workbooks. These tools are used broadly across Water Boards programs, and many are used to inform actions that could be considered consequential actions under the bill. To meet the bill’s provision, the State Water Board estimates significant cost pressures, likely in the millions of dollars per year. Costs would allow the State Water Board to fulfill several requirements regarding each of these covered systems. This would include facilitating and conducting a performance evaluation, contracting with an independent third party to perform audits on any systems the Water Boards develops, notifying those impacted by the consequential decision about the use of the ADS, and providing an appeal process. Taking these steps for each covered system, if feasible at all, would require significant additional staff and contract resources in the millions of dollars annually, impacting all Water Boards’ fund sources. Alternatively, State Water Board programs would need to find ways to store, process, and analyze data to make decisions without using any tool which would qualify as an automated decision system. Costs would be incurred from all eligible State Water Board fund sources.
- UC Health estimates a fiscal impact of $42 million in one-time costs and approximately $24 million in ongoing costs to implement AB 1018. Compliance costs for the University’s health systems include expenses related to implementing required audits, administrative processes, staff training, and system modifications.
- The California Department of Health and Human Services (CalHHS) reports a significant fiscal impact (General Fund, special funds). The Office of Technology and Solutions Integration (OTSI) estimates costs, across the 12 CalHHS departments, in the low millions up to low tens of millions (Total Fund) annually as solutions applicable under the bill are expected to grow in use over the coming years. These costs are estimated and attributable due to legal and security resources for monitoring and enforcement, compliance oversight, IT development and staffing necessary for governance and appeals management. This bill could impact many of the Agency's information systems and data solutions as Generative AI solutions are being built into many of the tools used. Evaluation and governance will become ongoing cost-pressure as it becomes necessary to review not just risk/compliance related to AI components of solutions, but applicability to the requirements of this bill. This bill would require programming and processing revisions, stakeholder engagement, development and operation of an appeal process in all cases, along with a third-party audit (beginning in 2030). The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the appeals processes that are yet to be determined. This estimate does not include potential increased contracting costs due to vendors including costs for conducting the assessment and other requirements.
- The California Department of Corrections and Rehabilitation (CDCR) reports that it does not currently use platforms it considers to be Automated Decision Systems (ADS). As a result, CDCR does not anticipate immediate costs associated with the requirements in AB 1018. However, should the department choose to implement or develop systems in the future that meet the bill’s definition of an ADS, there could be costs associated with compliance. These may include but are not limited to performance evaluations, third-party audits, disclosure and documentation requirements, and staff oversight responsibilities outlined in the bill. It should be noted that CDCR uses several risk assessment tools, including, but not limited to, the Static-99R to assess every eligible sex offender prior to release on parole, the California Static Risk Assessment tool to assess the risk of reoffending, the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) to assess an incarcerated individuals needs and target those needs, and the Automated Reentry Management System (ARMS). If this bill is interpreted to apply to these tools, CDCR, and other correctional agencies, could potentially incur a significant fiscal impact.
- The Department of Finance (DOF) reports potential unknown costs (General Fund) and indicates that that this bill may apply to its ongoing pilot using generative artificial intelligence to assist with bill analyses.
- Unknown, potentially significant costs to the Judicial Council, likely in the hundreds of millions annually. This bill could hamper the ability of trial courts and their justice partners to use several tools, including SARATSO for sex offenders, and many others, such as pretrial risk assessment tools minimize bias in pretrial detention situations. Developers of these tools, including nonprofits, universities, and even the state of Virginia, are unlikely to make any adjustments as would be required under this bill. As such, the entire judicial branch could potentially lose the use of all pretrial risk assessment tools, as their developers do not necessarily have a strong financial incentive to adhere to this bill’s stringent requirements. Courts also use tools such as MyCitation, which would also be affected. MyCitation is a tool created by the Judicial Council pursuant to AB 143 (Committee on Budget) Stats. 2021, ch. 79, designed as an easy-to-use option for defendants to provide relevant financial information for the court to consider reductions in infraction fines and fees owed.
- Unknown, potentially significant costs to the state funded trial court system (Trial Court Trust Fund, General Fund). By authorizing a new civil penalty with statutory damages, this bill will likely lead to additional case filings that otherwise would not have been commenced, with attendant workload and resource costs to the court. This bill may result in additional trial court costs to the extent that cases are brought to litigate its constitutionality. The fiscal impact of this bill to the courts will depend on many unknowns, including the number of cases filed and the factors unique to each case. An eight-hour court day costs approximately $10,500 in staff in workload. This is a conservative estimate, based on the hourly rate of court personnel including at minimum the judge, clerk, bailiff, court reporter, jury administrator, administrative staff, and jury per-diems. If court days exceed 10, costs to the trial courts could reach hundreds of thousands of dollars. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations.
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with this bill’s notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate, for all affected state agencies, costs may be in the hundreds of millions of dollars annually ongoing.
SUPPORT: (Verified 8/30/26)
TechEquity Action (source)
ACCE Action (Alliance of Californians for Community Empowerment)
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Civil Liberties Advocacy
California Community Foundation
California Employment Lawyers Association
California Federation of Labor Unions, AFL-CIO
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, AFT, AFL-CIO
Child Care Law Center
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Housing Now!
Justice2jobs Coalition
Kapor Center
LA Defensa
LAANE (Los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Mujeres Unidas Y Activas
Northern California District Council of the International Longshore and Warehouse Union
Oakland Privacy
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
SEIU California
Smart Justice California
Surveillance Resistance Lab
Tech Oversight Project
UDW/AFSCME Local 3930
UFCW - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
OPPOSITION: (Verified 8/30/26)
Advanced Medical Technology Association
Aerospace and Defense Alliance of California
American College of Obstetricians & Gynecologists - District Ix
American Financial Services Association
American Innovators Network
American Property Casualty Insurance Association
American Staffing Association
America's Physician Groups
Associated General Contractors of California
Association of California Life and Health Insurance Companies
Association of Independent California Colleges and Universities
Association of National Advertisers
Business Software Alliance
Business Software Association
Calbroadband
Calcom Association
California Association of Clerks & Election Officials
California Association of Health Plans
California Chamber of Commerce
California Community Banking Network
California Credit Union League
California Dental Association
California Department of Insurance
California Financial Services Association
California Fuels and Convenience Alliance
California Hospital Association
California Land Title Association
California Manufacturers and Technology Association
California Medical Association
California Mortgage Bankers Association
California Radiological Society
California Retailers Association
California Staffing Professionals
Chamber of Progress
Civil Justice Association of California
College Board
Computer and Communications Industry Association
Connected Commerce Council
Consumer Data Industry Association
Consumer Technology Association
Contra Costa County
County of Ventura
CTIA
Delta Dental of California
El Dorado County Chamber of Commerce
El Dorado Hills Chamber of Commerce
Elk Grove Chamber of Commerce
Family Business Association of California
Financial Technology Association
Folsom Chamber of Commerce
Hollywood Chamber of Commerce
Innovative Lending Platform Association
Internet Works
Internet.works
Judicial Council of California
Kaiser Permanente
Lincoln Chamber of Commerce
Los Angeles County
Los Angeles County Business Federation
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Ochin, INC.
Orange County Business Council
Personal Insurance Federation of California
Psinapse Techology
Public Risk Innovation, Solutions, and Management
Rancho Cordova Chamber of Commerce
Receivables Management Association International
Rocklin Area Chamber of Commerce
Roseville Area Chamber of Commerce
Santa Maria Valley Chamber of Commerce
Securities Industry and Financial Markets Association
Security Industry Association
Shingle Springs/cameron Park Chamber of Commerce
Silicon Valley Leadership Group
Society for Human Resource Management
Software Information Industry Association
STAR Staffing
Sutter Health
Techca
Technet
The Payments Coalition
United Chamber Advocacy Network Ucan
University of California
Valley Industry and Commerce Association
Yuba Sutter Chamber of Commerce
ARGUMENTS IN SUPPORT: A broad coalition, including the sponsor, TechEquity Action, write: While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile,[1] to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones.[2] Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives.
ARGUMENTS IN OPPOSITION: The Chamber of Progress writes:
While the recent amendments are a small step in the right direction, businesses that deploy covered automated decision systems would still be subject to significant costs in the form of time, energy, and money for the following: identify covered systems, determine which decisions and outcomes trigger the statute, implement pre-decision and adverse-outcome notices, establish correction and appeal processes, modify IT and operational workflows, train personnel, maintain documentation, and manage ongoing legal and compliance risk.
AB 1018 as written still mandates that online platforms inform any “natural person” who is affected by a consequential decision when an automated decision tool is employed to make such a determination. However, the bill does not define this term, despite referencing it continually for the sake of applicability. Such ambiguity risks stifling consumer welfare-enhancing innovations.
ASSEMBLY FLOOR: 50-16, 6/2/25
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Ávila Farías, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Caloza, Connolly, Elhawary, Fong, Gabriel, Garcia, Gipson, Mark González, Haney, Harabedian, Hart, Irwin, Jackson, Kalra, Krell, Lee, Lowenthal, McKinnor, Muratsuchi, Nguyen, Ortega, Papan, Patel, Pellerin, Quirk-Silva, Celeste Rodriguez, Rogers, Blanca Rubio, Schiavo, Schultz, Sharp-Collins, Solache, Stefani, Ward, Wicks, Wilson, Zbur, Rivas
NOES: Castillo, Chen, Davies, DeMaio, Dixon, Ellis, Gallagher, Jeff Gonzalez, Hadwick, Hoover, Lackey, Macedo, Patterson, Sanchez, Ta, Tangipa
NO VOTE RECORDED: Ahrens, Alanis, Bains, Carrillo, Flora, Pacheco, Petrie-Norris, Ramos, Ransom, Michelle Rodriguez, Soria, Valencia, Wallis
Prepared by: Christian Kurpiewski / P., D.T., & C.P. / (916) 651-1548
8/30/26 15:50:18
**** END ****
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-4171 | AB 1018 |
THIRD READING
Bill No: AB 1018
Author: Bauer-Kahan (D), et al.
Amended: 8/21/26 in Senate
Vote: 21
SENATE JUDICIARY COMMITTEE: 11-2, 7/15/25
AYES: Umberg, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Wahab, Weber Pierson, Wiener
NOES: Niello, Valladares
SENATE APPROPRIATIONS COMMITTEE: 5-2, 8/29/25
AYES: Caballero, Cabaldon, Grayson, Richardson, Wahab
NOES: Seyarto, Dahle
ASSEMBLY FLOOR: 50-16, 6/2/25 - See last page for vote
SUBJECT: Automated decision systems
SOURCE: Service Employees International Union California
TechEquity Action
DIGEST:
This bill regulates the use of “automated decision systems” (ADS). It places obligations on developers and deployers of such systems designed or used to make or facilitate “consequential decisions.”
Senate Floor amendments of 9/5/25 narrow the scope of this bill by, in part, carving out certain generative artificial intelligence (AI) systems and adding several exemptions. The amendments also reduce the burden on developers and deployers by, in part, reducing the document retention timeline and including factors for courts to consider in assessing civil penalties.
Senate Floor amendments of 8/21/26 add various exemptions and significantly narrow the bill by, among other things, removing requirements to conduct risk assessments and audits, and limiting certain rights to “adverse outcomes.”
ANALYSIS:
Existing law:
- Establishes the California Consumer Privacy Act (CCPA), which grants consumers certain rights with regard to their personal information, including enhanced notice, access, and disclosure; the right to deletion; the right to restrict the sale of information; and protection from discrimination for exercising these rights. It places attendant obligations on businesses to respect those rights. (Civil (Civ.) Code § 1798.100 et seq.)
- Establishes the Consumer Privacy Rights Act (CPRA), which amends the CCPA and creates the California Privacy Protection Agency (PPA), which is charged with implementing these laws, promulgating regulations, and carrying out enforcement actions. (Civ. Code § 1798.100 et seq.; Proposition 24 (2020).)
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Government (Gov.) Code § 12930.)
- Establishes the Fair Employment and Housing Act (FEHA). (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that meets specified criteria. (Civ. Code § 3426.1(d).)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms. (Gov. Code § 11546.45.5.)
This bill:
- Requires a developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer to provide the deployer with specified information.
- Dictates the process and form that any documentation required to be provided to a deployer by a developer must take.
- Provides that the above do not apply to a developer subject to the federal Gramm-Leach-Bliley Act who uses a covered ADS to make or facilitate a consequential decision related to financial services, as described.
- Requires a deployer, before it finalizes a consequential decision made or facilitated by a covered ADS, and after it finalizes such a decision that results in an adverse outcome, to provide any subject of that decision with specified information. A deployer must also provide the subject with an opportunity to correct any incorrect personal information used and to appeal the adverse consequential decision according to specified processes. Dictates the process and form that documentation required to be provided to a subject by a deployer must take.
- Provides certain exemptions from the above requirements for deployers subject to the federal Family Educational Rights and Privacy Act, the federal Equal Credit Opportunity Act, and the federal Fair Credit Reporting Act.
- Provides that a deployer’s collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
- Provides the conditions under which a deployer assumes the responsibilities of a developer.
- Imposes retention requirements on developers and deployers with respect to specified documentation.
- Allows for the redaction of certain information revealing trade secrets, as provided, but requires disclosure of such withholding and the basis for it.
- Authorizes enforcement actions against developers or deployers to be brought by specified public entities.
- Authorizes a court to award a prevailing plaintiff in such actions specified remedies, including a civil penalty of up to $25,000 per violation. It provides courts with relevant circumstances to be considered in assessing the civil penalty.
- Exempts specified ADS, entities regulated by HIPAA, and certain medical devices.
- Clarifies the relevance of compliance, or the failure to comply, with these provisions in actions pursuant to the Unruh Civil Rights Act and FEHA.
- Defines AI, ADS, trade secret, adverse outcome, and personal information in accordance with existing law and defines other relevant terms.
Background
ADS, especially those powered by AI, are being increasingly deployed in a multitude of contexts, including employment, housing, education, and health care. Major transparency and fairness concerns have been raised about the use of ADS to make consequential decisions, essentially determinations with significant legal or other material effect on people’s lives.
This bill seeks to regulate the development and deployment, by both public and private actors, of “covered ADS.” Developers are required to provide deployers certain information, and deployers are required to provide certain notices to subjects of consequential decisions and afford certain rights to them.
This bill is sponsored by TechEquity Action and the Service Employees International Union (SEIU) California. It is supported by a variety of labor, consumer, privacy, and other advocacy groups, including Common Sense Media and Consumer Reports. It is opposed by a number of industry associations, including the Consumer Technology Association and the American Staffing Association. For a more thorough discussion, please see the Senate Judiciary Committee analysis of this bill, which is incorporated by reference herein.
Comments
According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- Unknown, potentially significant costs (local funds, General Fund) to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the hundreds of millions of dollars annually statewide. For example, counties will be required to review all of their systems for compliance—including tools used for online testing, hiring, permitting, case management, risk assessments, and property management. They will also incur costs of audits, training, staffing, disclosures, appeals process, data retention, sending out notices, and potential legal exposure. If developers raise their cost as a result of this bill, these costs will be passed onto the counties. Counties may claim reimbursement of those costs if the Commission on State Mandates determines that this bill creates a new program or imposes a higher level of service on local agencies.
- The State Controller’s Office (SCO) reports that it, and all other state governmental entities would be considered a “deployer,” of an ADS, as defined in the bill. If SCO implemented one or multiple ADS systems that meet the criteria for oversight in AB 1018, SCO would anticipate needing the following General Fund Resources:
- 2025-26: $1,408,000 for 7.0 two-year limited-term positions and one-time costs;
- 2026-27: $1,384,000 for 7.0 two-year limited-term positions; and,
- 2027-28 and ongoing: $761,000 for 4.0 permanent positions.
SCO indicates that these estimates reflect the requirements for a large, complex agency such as the SCO managing sensitive personal and financial data across automated systems. Actual staffing may be adjusted depending on the complexity and scale of the deployed ADS. Some functions may overlap or be handled by existing staff with appropriate upskilling, while external contractors could augment capacity for short-term surges, such as during initial implementation or major audits. If AB 1018 becomes law, the SCO would need 5–7 information security staff—including a dedicated supervisor—for the initial implementation of AB 1018, with 3–4 (including the supervisor) required for ongoing security and compliance operations related to automated decision systems.
- Costs (special fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce Labor Code violations, possibly in the millions of dollars, annually. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- The Department of Justice (DOJ) estimates a fiscal impact of $2.5 million or less (General Fund, Unfair Competition Law Fund). The DOJ notes that implementation of this bill will be dependent upon the appropriation of funds. The DOJ will be unable to absorb the costs to comply with or implement the requirements of the bill within existing budgeted resources. The Consumer Protection Section (CPS), Civil Rights Enforcement Section (CRES) and Healthcare Rights & Access Section (HRA) within the Public Rights Division of the DOJ anticipate increased workloads in enforcing this bill beginning on January 1, 2026, and ongoing. The workload includes investigating and prosecuting violations. The Sections will require the following additional resources:
- CPS: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution;
- CRES: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution; and,
- HRA: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution.
- The Civil Rights Department (CRD) estimates a fiscal impact over $2 million (General Fund). CRD notes that this bill places requirements on ADS developers and deployers. It also provides the CRD with the authority to bring a civil action to enforce compliance with this bill. CRD anticipates requesting eight positions, including researchers/ research data specialists, associate governmental program analysts, and attorneys, approximately $2.2 million for FY 26-27, and $2.1 million for FY 27-28 and ongoing. In addition to covering the costs of the new positions, these resources would also fund an ongoing contract with an expert consultant, IT and other administrative costs, and a one-time job analysis that would be required for one of the research positions. These resources would be necessary to support the complex legal and technological analysis required to enforce the components of this bill.
- The State Water Board estimates a significant fiscal impact (General Fund, special funds) likely in the millions of dollars per year. The State Water Board reports, as written, this bill’s definition of automated decision system is vague, ambiguous, and could encompass many current tools used, like excel workbooks. These tools are used broadly across Water Boards programs, and many are used to inform actions that could be considered consequential actions under the bill. To meet the bill’s provision, the State Water Board estimates significant cost pressures, likely in the millions of dollars per year. Costs would allow the State Water Board to fulfill several requirements regarding each of these covered systems. This would include facilitating and conducting a performance evaluation, contracting with an independent third party to perform audits on any systems the Water Boards develops, notifying those impacted by the consequential decision about the use of the ADS, and providing an appeal process. Taking these steps for each covered system, if feasible at all, would require significant additional staff and contract resources in the millions of dollars annually, impacting all Water Boards’ fund sources. Alternatively, State Water Board programs would need to find ways to store, process, and analyze data to make decisions without using any tool which would qualify as an automated decision system. Costs would be incurred from all eligible State Water Board fund sources.
- UC Health estimates a fiscal impact of $42 million in one-time costs and approximately $24 million in ongoing costs to implement AB 1018. Compliance costs for the University’s health systems include expenses related to implementing required audits, administrative processes, staff training, and system modifications.
- The California Department of Health and Human Services (CalHHS) reports a significant fiscal impact (General Fund, special funds). The Office of Technology and Solutions Integration (OTSI) estimates costs, across the 12 CalHHS departments, in the low millions up to low tens of millions (Total Fund) annually as solutions applicable under the bill are expected to grow in use over the coming years. These costs are estimated and attributable due to legal and security resources for monitoring and enforcement, compliance oversight, IT development and staffing necessary for governance and appeals management. This bill could impact many of the Agency's information systems and data solutions as Generative AI solutions are being built into many of the tools used. Evaluation and governance will become ongoing cost-pressure as it becomes necessary to review not just risk/compliance related to AI components of solutions, but applicability to the requirements of this bill. This bill would require programming and processing revisions, stakeholder engagement, development and operation of an appeal process in all cases, along with a third-party audit (beginning in 2030). The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the appeals processes that are yet to be determined. This estimate does not include potential increased contracting costs due to vendors including costs for conducting the assessment and other requirements.
- The California Department of Corrections and Rehabilitation (CDCR) reports that it does not currently use platforms it considers to be Automated Decision Systems (ADS). As a result, CDCR does not anticipate immediate costs associated with the requirements in AB 1018. However, should the department choose to implement or develop systems in the future that meet the bill’s definition of an ADS, there could be costs associated with compliance. These may include but are not limited to performance evaluations, third-party audits, disclosure and documentation requirements, and staff oversight responsibilities outlined in the bill. It should be noted that CDCR uses several risk assessment tools, including, but not limited to, the Static-99R to assess every eligible sex offender prior to release on parole, the California Static Risk Assessment tool to assess the risk of reoffending, the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) to assess an incarcerated individuals needs and target those needs, and the Automated Reentry Management System (ARMS). If this bill is interpreted to apply to these tools, CDCR, and other correctional agencies, could potentially incur a significant fiscal impact.
- The Department of Finance (DOF) reports potential unknown costs (General Fund) and indicates that that this bill may apply to its ongoing pilot using generative artificial intelligence to assist with bill analyses.
- Unknown, potentially significant costs to the Judicial Council, likely in the hundreds of millions annually. This bill could hamper the ability of trial courts and their justice partners to use several tools, including SARATSO for sex offenders, and many others, such as pretrial risk assessment tools minimize bias in pretrial detention situations. Developers of these tools, including nonprofits, universities, and even the state of Virginia, are unlikely to make any adjustments as would be required under this bill. As such, the entire judicial branch could potentially lose the use of all pretrial risk assessment tools, as their developers do not necessarily have a strong financial incentive to adhere to this bill’s stringent requirements. Courts also use tools such as MyCitation, which would also be affected. MyCitation is a tool created by the Judicial Council pursuant to AB 143 (Committee on Budget) Stats. 2021, ch. 79, designed as an easy-to-use option for defendants to provide relevant financial information for the court to consider reductions in infraction fines and fees owed.
- Unknown, potentially significant costs to the state funded trial court system (Trial Court Trust Fund, General Fund). By authorizing a new civil penalty with statutory damages, this bill will likely lead to additional case filings that otherwise would not have been commenced, with attendant workload and resource costs to the court. This bill may result in additional trial court costs to the extent that cases are brought to litigate its constitutionality. The fiscal impact of this bill to the courts will depend on many unknowns, including the number of cases filed and the factors unique to each case. An eight-hour court day costs approximately $10,500 in staff in workload. This is a conservative estimate, based on the hourly rate of court personnel including at minimum the judge, clerk, bailiff, court reporter, jury administrator, administrative staff, and jury per-diems. If court days exceed 10, costs to the trial courts could reach hundreds of thousands of dollars. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations.
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with this bill’s notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate, for all affected state agencies, costs may be in the hundreds of millions of dollars annually ongoing.
SUPPORT: (Verified 8/22/26)
SEIU California (source)
TechEquity Action (source)
ACCE Action (Alliance of Californians for Community Empowerment)
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Civil Liberties Advocacy
California Community Foundation
California Employment Lawyers Association
California Federation of Labor Unions, AFL-CIO
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, AFT, AFL-CIO
Child Care Law Center
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Housing Now!
Justice2jobs Coalition
Kapor Center
LA Defensa
LAANE (Los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Mujeres Unidas Y Activas
Northern California District Council of the International Longshore and Warehouse Union
Oakland Privacy
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Tech Oversight Project
UDW/AFSCME Local 3930
UFCW - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
OPPOSITION: (Verified 8/22/26)
Advanced Medical Technology Association
Aerospace and Defense Alliance of California
American College of Obstetricians & Gynecologists - District Ix
American Financial Services Association
American Innovators Network
American Property Casualty Insurance Association
American Staffing Association
America's Physician Groups
Associated General Contractors of California
Association of California Life and Health Insurance Companies
Association of National Advertisers
Business Software Alliance
Business Software Association
Calbroadband
Calcom Association
California Association of Clerks & Election Officials
California Association of Health Plans
California Bankers Association
California Chamber of Commerce
California Community Banking Network
California Credit Union League
California Dental Association
California Financial Services Association
California Fuels and Convenience Alliance
California Hospital Association
California Land Title Association
California Life Sciences
California Manufacturers and Technology Association
California Medical Association
California Mortgage Bankers Association
California Radiological Society
California Retailers Association
California Staffing Professionals
Chamber of Progress
Civil Justice Association of California
College Board
Computer and Communications Industry Association
Connected Commerce Council
Consumer Data Industry Association
Consumer Technology Association
CTIA
Delta Dental of California
El Dorado County Chamber of Commerce
El Dorado Hills Chamber of Commerce
Elk Grove Chamber of Commerce
Family Business Association of California
Financial Technology Association
Fiserv
Folsom Chamber of Commerce
Hollywood Chamber of Commerce
Innovative Lending Platform Association
Internet Works
Internet.works
Judicial Council of California
Kaiser Permanente
Lincoln Chamber of Commerce
Los Angeles County
Los Angeles County Business Federation
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Ochin, INC.
Orange County Business Council
Personal Insurance Federation of California
Psinapse Techology
Public Risk Innovation, Solutions, and Management
Rancho Cordova Chamber of Commerce
Receivables Management Association International
Rocklin Area Chamber of Commerce
Roseville Area Chamber of Commerce
Santa Maria Valley Chamber of Commerce
Securities Industry and Financial Markets Association
Security Industry Association
Shingle Springs/cameron Park Chamber of Commerce
Silicon Valley Leadership Group
Society for Human Resource Management
Software Information Industry Association
STAR Staffing
Sutter Health
Techca
Technet
The Payments Coalition
United Chamber Advocacy Network Ucan
University of California
Valley Industry and Commerce Association
Yuba Sutter Chamber of Commerce
ARGUMENTS IN SUPPORT: A broad coalition, including the sponsors, SEIU California and TechEquity Action, write: While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile,[1] to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones.[2] Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives.
ARGUMENTS IN OPPOSITION: A coalition of financial industry groups, including the California Credit Union League, writes:
While Section 22756.2(b)(1) provides the authority to a financial institution to deny an opt-out request for financial services providers regulated under the Gramm Leach Bliley Act, the nation’s most robust law governing how financial information is used, it is unclear to us why there is not a similar exemption from the entirety of the bill. As written today, financial institutions could choose to categorically deny any consumer’s opt-out request. This will ultimately frustrate and confuse consumers, which is the opposite of the state intention of AB 1018 – which is to bring more transparency.
The Equal Credit Opportunities Act (ECOA) and Fair Credit Reporting Act (FCRA) already require creditors to provide a notice to consumers when they are denied credit detailing the reason for the denial and a method to dispute the accuracy or completeness of any information that led to the denial. Yet, financial services must still provide an opportunity to correct information, which is duplicative of FCRA, and an opportunity to appeal, which can impact the lending process. Establishing a consumer appeal provision would require a type of manual underwriting that has not been regularly used in the mortgage industry for nearly three decades. The current mortgage origination process relies on automated systems such as AUSs and Credit Scoring Models (CSMs) that do not have usable or scalable alternative manual processes.
ASSEMBLY FLOOR: 50-16, 6/2/25
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Ávila Farías, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Caloza, Connolly, Elhawary, Fong, Gabriel, Garcia, Gipson, Mark González, Haney, Harabedian, Hart, Irwin, Jackson, Kalra, Krell, Lee, Lowenthal, McKinnor, Muratsuchi, Nguyen, Ortega, Papan, Patel, Pellerin, Quirk-Silva, Celeste Rodriguez, Rogers, Blanca Rubio, Schiavo, Schultz, Sharp-Collins, Solache, Stefani, Ward, Wicks, Wilson, Zbur, Rivas
NOES: Castillo, Chen, Davies, DeMaio, Dixon, Ellis, Gallagher, Jeff Gonzalez, Hadwick, Hoover, Lackey, Macedo, Patterson, Sanchez, Ta, Tangipa
NO VOTE RECORDED: Ahrens, Alanis, Bains, Carrillo, Flora, Pacheco, Petrie-Norris, Ramos, Ransom, Michelle Rodriguez, Soria, Valencia, Wallis
Prepared by: Christian Kurpiewski / P., D.T., & C.P. / (916) 651-1548
8/24/26 12:02:06
**** END ****
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | AB 1018 |
THIRD READING
Bill No: AB 1018
Author: Bauer-Kahan (D), et al.
Amended: 9/5/25 in Senate
Vote: 21
SENATE JUDICIARY COMMITTEE: 11-2, 7/15/25
AYES: Umberg, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Wahab, Weber Pierson, Wiener
NOES: Niello, Valladares
SENATE APPROPRIATIONS COMMITTEE: 5-2, 8/29/25
AYES: Caballero, Cabaldon, Grayson, Richardson, Wahab
NOES: Seyarto, Dahle
ASSEMBLY FLOOR: 50-16, 6/2/25 - See last page for vote
SUBJECT: Automated decision systems
SOURCE: Service Employees International Union California
TechEquity Action
DIGEST:
This bill regulates the use of “automated decision systems” (ADS). It places obligations on developers and deployers of such systems designed or used to make or facilitate “consequential decisions.”
Senate Floor amendments of 9/5/25 narrow the scope of the bill by, in part, carving out generative AI systems and adding several exemptions. The amendments also reduce the burden on developers and deployers by, in part, reducing the document retention timeline and including factors for courts to consider in assessing civil penalties.
ANALYSIS:
Existing law:
- Establishes the California Consumer Privacy Act (CCPA), which grants consumers certain rights with regard to their personal information, including enhanced notice, access, and disclosure; the right to deletion; the right to restrict the sale of information; and protection from discrimination for exercising these rights. It places attendant obligations on businesses to respect those rights. (Civil (Civ.) Code § 1798.100 et seq.)
- Establishes the Consumer Privacy Rights Act (CPRA), which amends the CCPA and creates the California Privacy Protection Agency (PPA), which is charged with implementing these laws, promulgating regulations, and carrying out enforcement actions. (Civ. Code § 1798.100 et seq.; Proposition 24 (2020).)
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Government (Gov.) Code § 12930.)
- Establishes the Fair Employment and Housing Act (FEHA). (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that meets specified criteria. (Civ. Code § 3426.1(d).)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms. (Gov. Code § 11546.45.5.)
This bill:
- Requires a developer of a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, to conduct an initial impact assessment on the covered ADS before January 1, 2027 and annually thereafter. For covered ADS first deployed or made available on or after January 1, 2026, the developer shall conduct an impact assessment before initially deploying it or making it available to potential deployers, and annually thereafter.
- Requires a developer, in conducting a impact assessment, to carry out a number of specified duties.
- Requires a developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer to provide the deployer with specified information.
- Permits a developer to make reasonable redactions from the documentation provided to protect trade secrets, but requires notice of such withholding and the basis for it.
- Dictates the process and form that any documentation required to be provided to a deployer by a developer must take.
- Makes it unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent impact assessment conducted on the covered ADS.
- Requires developers to designate at least one employee to oversee the developer’s compliance herewith and to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
- Provides that the above do not apply to a developer subject to the federal Gramm-Leach-Bliley Act who uses a covered ADS to make or facilitate a consequential decision related to financial services, as described.
- Requires a deployer, before and after it finalizes a consequential decision made or facilitated by a covered ADS, to provide any subject of that decision with specified information. A deployer must also provide the subject with an opportunity to correct any incorrect personal information used and to appeal the decision according to specified processes. Dictates the process and form that documentation required to be provided to a subject by a deployer must take.
- Provides that a deployer’s collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
- Provides the conditions under which a deployer assumes the responsibilities of a developer.
- Requires a deployer that uses a covered ADS to make or facilitate a consequential decision to designate at least one employee to oversee the deployer’s compliance herewith. That employee must conduct a prompt and comprehensive review of any credible compliance issue related to the deployer’s use of a covered ADS that is raised to that employee.
- Requires a developer to provide, where applicable, an auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive audit. Provides that necessary audits not completed by the specified deadlines require the developer to not deploy or make available the ADS.
- Imposes retention requirements on developers and deployers with respect to specified documentation.
- Provides that the provisions governing deployers become operative on January 1, 2027.
- Allows for the redaction of information revealing trade secrets, as provided, but requires disclosure of such withholding and the basis for it.
- Provides that within 30 days of receiving a request from the Attorney General (AG) for a impact assessment, a developer or auditor of a covered ADS shall provide an unredacted copy of the document to the AG. The AG is authorized to share these materials with other enforcement entities as necessary for enforcement purposes.
- Clarifies that the above disclosure or sharing of a impact assessment does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained therein. Such materials are exempt from the California Public Records Act.
- Authorizes enforcement actions against developers, deployers, or auditors to be brought by specified public entities.
- Authorizes a court to award a prevailing plaintiff in such actions specified remedies, including a civil penalty of up to $25,000 per violation. It provides courts with relevant circumstances to be considered in assessing the civil penalty.
- Exempts specified ADS.
- Clarifies the relevance of compliance, or the failure to comply, with these provisions in actions pursuant to the Unruh Civil Rights Act and the Fair Employment and Housing Act.
- Defines AI, ADS, trade secret, and personal information in accordance with existing law and defines other relevant terms.
Background
ADS, especially those powered by AI, are being increasingly deployed in a multitude of contexts, including employment, housing, education, and health care. Major transparency and fairness concerns have been raised about the use of ADS to make consequential decisions, essentially determinations with significant legal or other material effect on people’s lives.
This bill seeks to regulate the development and deployment, by both public and private actors, of “covered ADS.” It requires developers to conduct impact assessments of their ADS, which, among other things, identify details of the systems, expected performance and uses, and potential disparate impacts. Deployers are required to provide certain notices to subjects of consequential decisions and afford certain rights to them.
This bill is sponsored by TechEquity Action and the Service Employees International Union (SEIU) California. It is supported by a variety of labor, consumer, privacy, and other advocacy groups, including Common Sense Media and Consumer Reports. It is opposed by a number of industry associations, including the Consumer Technology Association and the American Staffing Association. For a more thorough discussion, please see the Senate Judiciary Committee analysis of this bill, which is incorporated by reference herein.
Comments
According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- Unknown, potentially significant costs (local funds, General Fund) to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the hundreds of millions of dollars annually statewide. For example, counties will be required to review all of their systems for compliance—including tools used for online testing, hiring, permitting, case management, risk assessments, and property management. They will also incur costs of audits, training, staffing, disclosures, appeals process, data retention, sending out notices, and potential legal exposure. If developers raise their cost as a result of this bill, these costs will be passed onto the counties. Counties may claim reimbursement of those costs if the Commission on State Mandates determines that this bill creates a new program or imposes a higher level of service on local agencies.
- The State Controller’s Office (SCO) reports that it, and all other state governmental entities would be considered a “deployer,” of an ADS, as defined in the bill. If SCO implemented one or multiple ADS systems that meet the criteria for oversight in AB 1018, SCO would anticipate needing the following General Fund Resources:
- 2025-26: $1,408,000 for 7.0 two-year limited-term positions and one-time costs;
- 2026-27: $1,384,000 for 7.0 two-year limited-term positions; and,
- 2027-28 and ongoing: $761,000 for 4.0 permanent positions.
SCO indicates that these estimates reflect the requirements for a large, complex agency such as the SCO managing sensitive personal and financial data across automated systems. Actual staffing may be adjusted depending on the complexity and scale of the deployed ADS. Some functions may overlap or be handled by existing staff with appropriate upskilling, while external contractors could augment capacity for short-term surges, such as during initial implementation or major audits. If AB 1018 becomes law, the SCO would need 5–7 information security staff—including a dedicated supervisor—for the initial implementation of AB 1018, with 3–4 (including the supervisor) required for ongoing security and compliance operations related to automated decision systems.
- Costs (special fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce Labor Code violations, possibly in the millions of dollars, annually. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- The Department of Justice (DOJ) estimates a fiscal impact of $2.5 million or less (General Fund, Unfair Competition Law Fund). The DOJ notes that implementation of this bill will be dependent upon the appropriation of funds. The DOJ will be unable to absorb the costs to comply with or implement the requirements of the bill within existing budgeted resources. The Consumer Protection Section (CPS), Civil Rights Enforcement Section (CRES) and Healthcare Rights & Access Section (HRA) within the Public Rights Division of the DOJ anticipate increased workloads in enforcing this bill beginning on January 1, 2026, and ongoing. The workload includes investigating and prosecuting violations. The Sections will require the following additional resources:
- CPS: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution;
- CRES: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution; and,
- HRA: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution.
- The Civil Rights Department (CRD) estimates a fiscal impact over $2 million (General Fund). CRD notes that this bill places requirements on ADS developers and deployers. It also provides the CRD with the authority to bring a civil action to enforce compliance with this bill. CRD anticipates requesting eight positions, including researchers/ research data specialists, associate governmental program analysts, and attorneys, approximately $2.2 million for FY 26-27, and $2.1 million for FY 27-28 and ongoing. In addition to covering the costs of the new positions, these resources would also fund an ongoing contract with an expert consultant, IT and other administrative costs, and a one-time job analysis that would be required for one of the research positions. These resources would be necessary to support the complex legal and technological analysis required to enforce the components of this bill.
- The State Water Board estimates a significant fiscal impact (General Fund, special funds) likely in the millions of dollars per year. The State Water Board reports, as written, this bill’s definition of automated decision system is vague, ambiguous, and could encompass many current tools used, like excel workbooks. These tools are used broadly across Water Boards programs, and many are used to inform actions that could be considered consequential actions under the bill. To meet the bill’s provision, the State Water Board estimates significant cost pressures, likely in the millions of dollars per year. Costs would allow the State Water Board to fulfill several requirements regarding each of these covered systems. This would include facilitating and conducting a performance evaluation, contracting with an independent third party to perform audits on any systems the Water Boards develops, notifying those impacted by the consequential decision about the use of the ADS, and providing an appeal process. Taking these steps for each covered system, if feasible at all, would require significant additional staff and contract resources in the millions of dollars annually, impacting all Water Boards’ fund sources. Alternatively, State Water Board programs would need to find ways to store, process, and analyze data to make decisions without using any tool which would qualify as an automated decision system. Costs would be incurred from all eligible State Water Board fund sources.
- UC Health estimates a fiscal impact of $42 million in one-time costs and approximately $24 million in ongoing costs to implement AB 1018. Compliance costs for the University’s health systems include expenses related to implementing required audits, administrative processes, staff training, and system modifications.
- The California Department of Health and Human Services (CalHHS) reports a significant fiscal impact (General Fund, special funds). The Office of Technology and Solutions Integration (OTSI) estimates costs, across the 12 CalHHS departments, in the low millions up to low tens of millions (Total Fund) annually as solutions applicable under the bill are expected to grow in use over the coming years. These costs are estimated and attributable due to legal and security resources for monitoring and enforcement, compliance oversight, IT development and staffing necessary for governance and appeals management. This bill could impact many of the Agency's information systems and data solutions as Generative AI solutions are being built into many of the tools used. Evaluation and governance will become ongoing cost-pressure as it becomes necessary to review not just risk/compliance related to AI components of solutions, but applicability to the requirements of this bill. This bill would require programming and processing revisions, stakeholder engagement, development and operation of an appeal process in all cases, along with a third-party audit (beginning in 2030). The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the appeals processes that are yet to be determined. This estimate does not include potential increased contracting costs due to vendors including costs for conducting the assessment and other requirements.
- The California Department of Corrections and Rehabilitation (CDCR) reports that it does not currently use platforms it considers to be Automated Decision Systems (ADS). As a result, CDCR does not anticipate immediate costs associated with the requirements in AB 1018. However, should the department choose to implement or develop systems in the future that meet the bill’s definition of an ADS, there could be costs associated with compliance. These may include but are not limited to performance evaluations, third-party audits, disclosure and documentation requirements, and staff oversight responsibilities outlined in the bill. It should be noted that CDCR uses several risk assessment tools, including, but not limited to, the Static-99R to assess every eligible sex offender prior to release on parole, the California Static Risk Assessment tool to assess the risk of reoffending, the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) to assess an incarcerated individuals needs and target those needs, and the Automated Reentry Management System (ARMS). If this bill is interpreted to apply to these tools, CDCR, and other correctional agencies, could potentially incur a significant fiscal impact.
- The Department of Finance (DOF) reports potential unknown costs (General Fund) and indicates that that this bill may apply to its ongoing pilot using generative artificial intelligence to assist with bill analyses.
- Unknown, potentially significant costs to the Judicial Council, likely in the hundreds of millions annually. This bill could hamper the ability of trial courts and their justice partners to use several tools, including SARATSO for sex offenders, and many others, such as pretrial risk assessment tools minimize bias in pretrial detention situations. Developers of these tools, including nonprofits, universities, and even the state of Virginia, are unlikely to make any adjustments as would be required under this bill. As such, the entire judicial branch could potentially lose the use of all pretrial risk assessment tools, as their developers do not necessarily have a strong financial incentive to adhere to this bill’s stringent requirements. Courts also use tools such as MyCitation, which would also be affected. MyCitation is a tool created by the Judicial Council pursuant to AB 143 (Committee on Budget) Stats. 2021, ch. 79, designed as an easy-to-use option for defendants to provide relevant financial information for the court to consider reductions in infraction fines and fees owed.
- Unknown, potentially significant costs to the state funded trial court system (Trial Court Trust Fund, General Fund). By authorizing a new civil penalty with statutory damages, this bill will likely lead to additional case filings that otherwise would not have been commenced, with attendant workload and resource costs to the court. This bill may result in additional trial court costs to the extent that cases are brought to litigate its constitutionality. The fiscal impact of this bill to the courts will depend on many unknowns, including the number of cases filed and the factors unique to each case. An eight-hour court day costs approximately $10,500 in staff in workload. This is a conservative estimate, based on the hourly rate of court personnel including at minimum the judge, clerk, bailiff, court reporter, jury administrator, administrative staff, and jury per-diems. If court days exceed 10, costs to the trial courts could reach hundreds of thousands of dollars. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations.
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with this bill’s notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate, for all affected state agencies, costs may be in the hundreds of millions of dollars annually ongoing.
SUPPORT: (Verified 9/8/25)
SEIU California (source)
TechEquity Action (source)
ACCE Action (Alliance of Californians for Community Empowerment)
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Civil Liberties Advocacy
California Community Foundation
California Employment Lawyers Association
California Federation of Labor Unions, AFL-CIO
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, AFT, AFL-CIO
Child Care Law Center
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Housing Now!
Justice2jobs Coalition
Kapor Center
LA Defensa
LAANE (Los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Mujeres Unidas Y Activas
Northern California District Council of the International Longshore and Warehouse Union
Oakland Privacy
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Tech Oversight Project
UDW/AFSCME Local 3930
UFCW - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
OPPOSITION: (Verified 9/8/25)
Advanced Medical Technology Association
Aerospace and Defense Alliance of California
American College of Obstetricians & Gynecologists - District Ix
American Financial Services Association
American Innovators Network
American Property Casualty Insurance Association
American Staffing Association
America's Physician Groups
Associated General Contractors of California
Association of California Life and Health Insurance Companies
Association of National Advertisers
Business Software Alliance
Business Software Association
Calbroadband
Calcom Association
California Association of Clerks & Election Officials
California Association of Health Plans
California Bankers Association
California Chamber of Commerce
California Community Banking Network
California Credit Union League
California Dental Association
California Financial Services Association
California Fuels and Convenience Alliance
California Hospital Association
California Land Title Association
California Life Sciences
California Manufacturers and Technology Association
California Medical Association (CMA)
California Mortgage Bankers Association
California Radiological Society
California Retailers Association
California Staffing Professionals (CSP)
Chamber of Progress
Civil Justice Association of California (CJAC)
College Board
Computer and Communications Industry Association
Connected Commerce Council
Consumer Data Industry Association
Consumer Technology Association
CTIA
Delta Dental of California
El Dorado County Chamber of Commerce
El Dorado Hills Chamber of Commerce
Elk Grove Chamber of Commerce
Family Business Association of California
Financial Technology Association
Fiserv
Folsom Chamber of Commerce
Hollywood Chamber of Commerce
Innovative Lending Platform Association
Internet Works
Internet.works
Judicial Council of California
Kaiser Permanente
Lincoln Chamber of Commerce
Los Angeles County
Los Angeles County Business Federation
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Ochin, INC.
Orange County Business Council
Personal Insurance Federation of California
Psinapse Techology
Public Risk Innovation, Solutions, and Management (PRISM)
Rancho Cordova Chamber of Commerce
Receivables Management Association International
Rocklin Area Chamber of Commerce
Roseville Area Chamber of Commerce
Santa Maria Valley Chamber of Commerce
Securities Industry and Financial Markets Association
Security Industry Association
Shingle Springs/cameron Park Chamber of Commerce
Silicon Valley Leadership Group
Society for Human Resource Management
Software Information Industry Association
STAR Staffing
Sutter Health
Techca
Technet
The Payments Coalition
United Chamber Advocacy Network Ucan
University of California
Valley Industry and Commerce Association (VICA)
Yuba Sutter Chamber of Commerce
ARGUMENTS IN SUPPORT: A broad coalition, including the sponsors, SEIU California and TechEquity Action, write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile,[1] to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones.[2] Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives.
ARGUMENTS IN OPPOSITION: A coalition of financial industry groups, including the California Credit Union League, writes:
While Section 22756.2(b)(1) provides the authority to a financial institution to deny an opt-out request for financial services providers regulated under the Gramm Leach Bliley Act, the nation’s most robust law governing how financial information is used, it is unclear to us why there is not a similar exemption from the entirety of the bill. As written today, financial institutions could choose to categorically deny any consumer’s opt-out request. This will ultimately frustrate and confuse consumers, which is the opposite of the state intention of AB 1018 – which is to bring more transparency.
The Equal Credit Opportunities Act (ECOA) and Fair Credit Reporting Act (FCRA) already require creditors to provide a notice to consumers when they are denied credit detailing the reason for the denial and a method to dispute the accuracy or completeness of any information that led to the denial. Yet, financial services must still provide an opportunity to correct information, which is duplicative of FCRA, and an opportunity to appeal, which can impact the lending process. Establishing a consumer appeal provision would require a type of manual underwriting that has not been regularly used in the mortgage industry for nearly three decades. The current mortgage origination process relies on automated systems such as AUSs and Credit Scoring Models (CSMs) that do not have usable or scalable alternative manual processes.
ASSEMBLY FLOOR: 50-16, 6/2/25
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Ávila Farías, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Caloza, Connolly, Elhawary, Fong, Gabriel, Garcia, Gipson, Mark González, Haney, Harabedian, Hart, Irwin, Jackson, Kalra, Krell, Lee, Lowenthal, McKinnor, Muratsuchi, Nguyen, Ortega, Papan, Patel, Pellerin, Quirk-Silva, Celeste Rodriguez, Rogers, Blanca Rubio, Schiavo, Schultz, Sharp-Collins, Solache, Stefani, Ward, Wicks, Wilson, Zbur, Rivas
NOES: Castillo, Chen, Davies, DeMaio, Dixon, Ellis, Gallagher, Jeff Gonzalez, Hadwick, Hoover, Lackey, Macedo, Patterson, Sanchez, Ta, Tangipa
NO VOTE RECORDED: Ahrens, Alanis, Bains, Carrillo, Flora, Pacheco, Petrie-Norris, Ramos, Ransom, Michelle Rodriguez, Soria, Valencia, Wallis
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
9/9/25 10:16:46
**** END ****
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | AB 1018 |
THIRD READING
Bill No: AB 1018
Author: Bauer-Kahan (D), et al.
Amended: 7/17/25 in Senate
Vote: 21
SENATE JUDICIARY COMMITTEE: 11-2, 7/15/25
AYES: Umberg, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Wahab, Weber Pierson, Wiener
NOES: Niello, Valladares
SENATE APPROPRIATIONS COMMITTEE: 5-2, 8/29/25
AYES: Caballero, Cabaldon, Grayson, Richardson, Wahab
NOES: Seyarto, Dahle
ASSEMBLY FLOOR: 50-16, 6/2/25 - See last page for vote
SUBJECT: Automated decision systems
SOURCE: Service Employees International Union (SEIU) California
TechEquity Action
DIGEST:
This bill regulates the use of “automated decision systems” (ADS). It places obligations on developers and deployers of such systems designed or used to make or facilitate “consequential decisions.”
ANALYSIS:
Existing law:
- Establishes the California Consumer Privacy Act (CCPA), which grants consumers certain rights with regard to their personal information, including enhanced notice, access, and disclosure; the right to deletion; the right to restrict the sale of information; and protection from discrimination for exercising these rights. It places attendant obligations on businesses to respect those rights. (Civil (Civ.) Code § 1798.100 et seq.)
- Establishes the Consumer Privacy Rights Act (CPRA), which amends the CCPA and creates the California Privacy Protection Agency (PPA), which is charged with implementing these laws, promulgating regulations, and carrying out enforcement actions. (Civ. Code § 1798.100 et seq.; Proposition 24 (2020).)
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Government (Gov.) Code § 12930.)
- Establishes the Fair Employment and Housing Act (FEHA). (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that meets specified criteria. (Civ. Code § 3426.1(d).)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms. (Gov. Code § 11546.45.5.)
This bill:
- Requires a developer of a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, to conduct an initial performance evaluation on the covered ADS before January 1, 2027 and annually thereafter. For covered ADS first deployed or made available on or after January 1, 2026, the developer shall conduct a performance evaluation before initially deploying it or making it available to potential deployers, and annually thereafter.
- Requires a developer, in conducting a performance evaluation, to carry out a number of specified duties.
- Requires a developer to consider and attempt to incorporate any feedback from an auditor into the development of any subsequent version of the covered ADS and to make a high-level summary of the feedback publicly available.
- Requires a developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer to provide the deployer with specified information.
- Permits a developer to make reasonable redactions from the documentation provided to protect trade secrets, but requires notice of such withholding and the basis for it.
- Requires a developer that receives an impact assessment from an auditor of a deployed covered ADS to provide specified information to any deployer of the covered ADS. Dictates the process and form that any documentation required to be provided to a deployer by a developer must take.
- Makes it unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
- Requires developers to designate at least one employee to oversee the developer’s compliance herewith and to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
- Provides that the above do not apply to a developer subject to the federal Gramm-Leach-Bliley Act who uses a covered ADS to make or facilitate a consequential decision related to financial services, as described.
- Requires a deployer, before and after it finalizes a consequential decision made or facilitated by a covered ADS, to provide any subject of that decision with specified information. A deployer must also provide the subject with an opportunity to correct any incorrect personal information used and to appeal the decision according to specified processes. Dictates the process and form that documentation required to be provided to a subject by a deployer must take.
- Provides that a deployer’s collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
- Provides the conditions under which a deployer assumes the responsibilities of a developer.
- Requires a deployer that uses a covered ADS to make or facilitate a consequential decision to designate at least one employee to oversee the deployer’s compliance herewith. That employee must conduct a prompt and comprehensive review of any credible compliance issue related to the deployer’s use of a covered ADS that is raised to that employee.
- Requires a developer to provide, where applicable, an auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive audit. Provides that necessary audits not completed by the specified deadlines require the developer to not deploy or make available the ADS.
- Imposes retention requirements on developers and deployers with respect to specified documentation.
- Provides that the provisions governing deployers become operative on January 1, 2027.
- Allows for the redaction of information revealing trade secrets, as provided, but requires disclosure of such withholding and the basis for it.
- Provides that within 30 days of receiving a request from the Attorney General (AG) for a performance evaluation, a developer or auditor of a covered ADS shall provide an unredacted copy of the document to the AG. The AG is authorized to share these materials with other enforcement entities as necessary for enforcement purposes.
- Clarifies that the above disclosure or sharing of a performance evaluation does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained therein. Such materials are exempt from the California Public Records Act.
- Authorizes enforcement actions against developers, deployers, or auditors to be brought by specified public entities:
- Authorizes a court to award a prevailing plaintiff in such actions specified remedies, including a civil penalty of up to $25,000 per violation.
- Exempts ADS whose sole purpose is to detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems or to operate aircraft in the national airspace. The use of a consumer credit score to inform a consequential decision does not itself create an obligation hereunder.
- Clarifies the relevance of compliance, or the failure to comply, with these provisions in actions pursuant to the Unruh Civil Rights Act and the Fair Employment and Housing Act.
- Defines AI, ADS, trade secret, and personal information in accordance with existing law and defines other relevant terms.
Background
ADS, especially those powered by AI, are being increasingly deployed in a multitude of contexts, including employment, housing, education, and health care. Major transparency and fairness concerns have been raised about the use of ADS to make consequential decisions, essentially determinations with significant legal or other material effect on people’s lives.
This bill seeks to regulate the development and deployment, by both public and private actors, of “covered ADS,” systems that make consequential decisions, or decisions that materially impact the cost, terms, quality, or accessibility of specified services, opportunities, access, resources, and treatment for natural persons. It requires developers to conduct performance evaluations of their ADS, which, among other things, identify details of the systems, expected performance and uses, and potential disparate impacts. Deployers are required to provide certain notices to subjects of consequential decisions and afford certain rights to them, including the right to correct information and appeal final decisions.
This bill is sponsored by TechEquity Action and the Service Employees International Union (SEIU) California. It is supported by a variety of labor, consumer, privacy, and other advocacy groups, including Common Sense Media and Consumer Reports. It is opposed by a number of industry associations, including the Consumer Technology Association and the American Staffing Association. For a more thorough discussion, please see the Senate Judiciary Committee analysis of the bill, which is incorporated by reference herein.
Comment
According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- Unknown, potentially significant costs (local funds, General Fund) to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the hundreds of millions of dollars annually statewide. For example, counties will be required to review all of their systems for compliance—including tools used for online testing, hiring, permitting, case management, risk assessments, and property management. They will also incur costs of audits, training, staffing, disclosures, appeals process, data retention, sending out notices, and potential legal exposure. If developers raise their cost as a result of this bill, these costs will be passed onto the counties. Counties may claim reimbursement of those costs if the Commission on State Mandates determines that this bill creates a new program or imposes a higher level of service on local agencies.
- The State Controller’s Office (SCO) reports that it, and all other state governmental entities would be considered a “deployer,” of an ADS, as defined in the bill. If SCO implemented one or multiple ADS systems that meet the criteria for oversight in AB 1018, SCO would anticipate needing the following General Fund Resources:
- 2025-26: $1,408,000 for 7.0 two-year limited-term positions and one-time costs;
- 2026-27: $1,384,000 for 7.0 two-year limited-term positions; and,
- 2027-28 and ongoing: $761,000 for 4.0 permanent positions.
SCO indicates that these estimates reflect the requirements for a large, complex agency such as the SCO managing sensitive personal and financial data across automated systems. Actual staffing may be adjusted depending on the complexity and scale of the deployed ADS. Some functions may overlap or be handled by existing staff with appropriate upskilling, while external contractors could augment capacity for short-term surges, such as during initial implementation or major audits. If AB 1018 becomes law, the SCO would need 5–7 information security staff—including a dedicated supervisor—for the initial implementation of AB 1018, with 3–4 (including the supervisor) required for ongoing security and compliance operations related to automated decision systems.
- Costs (special fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce Labor Code violations, possibly in the millions of dollars, annually. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- The Department of Justice (DOJ) estimates a fiscal impact of $2.5 million or less (General Fund, Unfair Competition Law Fund). The DOJ notes that implementation of this bill will be dependent upon the appropriation of funds. The DOJ will be unable to absorb the costs to comply with or implement the requirements of the bill within existing budgeted resources. The Consumer Protection Section (CPS), Civil Rights Enforcement Section (CRES) and Healthcare Rights & Access Section (HRA) within the Public Rights Division of the DOJ anticipate increased workloads in enforcing this bill beginning on January 1, 2026, and ongoing. The workload includes investigating and prosecuting violations. The Sections will require the following additional resources:
- CPS: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution;
- CRES: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution; and,
- HRA: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution.
- The Civil Rights Department (CRD) estimates a fiscal impact over $2 million (General Fund). CRD notes that this bill places requirements on ADS developers and deployers. It also provides the CRD with the authority to bring a civil action to enforce compliance with this bill. CRD anticipates requesting eight positions, including researchers/ research data specialists, associate governmental program analysts, and attorneys, approximately $2.2 million for FY 26-27, and $2.1 million for FY 27-28 and ongoing. In addition to covering the costs of the new positions, these resources would also fund an ongoing contract with an expert consultant, IT and other administrative costs, and a one-time job analysis that would be required for one of the research positions. These resources would be necessary to support the complex legal and technological analysis required to enforce the components of this bill.
- The State Water Board estimates a significant fiscal impact (General Fund, special funds) likely in the millions of dollars per year. The State Water Board reports, as written, this bill’s definition of automated decision system is vague, ambiguous, and could encompass many current tools used, like excel workbooks. These tools are used broadly across Water Boards programs, and many are used to inform actions that could be considered consequential actions under the bill. To meet the bill’s provision, the State Water Board estimates significant cost pressures, likely in the millions of dollars per year. Costs would allow the State Water Board to fulfill several requirements regarding each of these covered systems. This would include facilitating and conducting a performance evaluation, contracting with an independent third party to perform audits on any systems the Water Boards develops, notifying those impacted by the consequential decision about the use of the ADS, and providing an appeal process. Taking these steps for each covered system, if feasible at all, would require significant additional staff and contract resources in the millions of dollars annually, impacting all Water Boards’ fund sources. Alternatively, State Water Board programs would need to find ways to store, process, and analyze data to make decisions without using any tool which would qualify as an automated decision system. Costs would be incurred from all eligible State Water Board fund sources.
- UC Health estimates a fiscal impact of $42 million in one-time costs and approximately $24 million in ongoing costs to implement AB 1018. Compliance costs for the University’s health systems include expenses related to implementing required audits, administrative processes, staff training, and system modifications.
- The California Department of Health and Human Services (CalHHS) reports a significant fiscal impact (General Fund, special funds). The Office of Technology and Solutions Integration (OTSI) estimates costs, across the 12 CalHHS departments, in the low millions up to low tens of millions (Total Fund) annually as solutions applicable under the bill are expected to grow in use over the coming years. These costs are estimated and attributable due to legal and security resources for monitoring and enforcement, compliance oversight, IT development and staffing necessary for governance and appeals management. This bill could impact many of the Agency's information systems and data solutions as Generative AI solutions are being built into many of the tools used. Evaluation and governance will become ongoing cost-pressure as it becomes necessary to review not just risk/compliance related to AI components of solutions, but applicability to the requirements of this bill. This bill would require programming and processing revisions, stakeholder engagement, development and operation of an appeal process in all cases, along with a third-party audit (beginning in 2030). The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the appeals processes that are yet to be determined. This estimate does not include potential increased contracting costs due to vendors including costs for conducting the assessment and other requirements.
- The California Department of Corrections and Rehabilitation (CDCR) reports that it does not currently use platforms it considers to be Automated Decision Systems (ADS). As a result, CDCR does not anticipate immediate costs associated with the requirements in AB 1018. However, should the department choose to implement or develop systems in the future that meet the bill’s definition of an ADS, there could be costs associated with compliance. These may include but are not limited to performance evaluations, third-party audits, disclosure and documentation requirements, and staff oversight responsibilities outlined in the bill. It should be noted that CDCR uses several risk assessment tools, including, but not limited to, the Static-99R to assess every eligible sex offender prior to release on parole, the California Static Risk Assessment tool to assess the risk of reoffending, the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) to assess an incarcerated individuals needs and target those needs, and the Automated Reentry Management System (ARMS). If this bill is interpreted to apply to these tools, CDCR, and other correctional agencies, could potentially incur a significant fiscal impact.
- The Department of Finance (DOF) reports potential unknown costs (General Fund) and indicates that that this bill may apply to its ongoing pilot using generative artificial intelligence to assist with bill analyses.
- Unknown, potentially significant costs to the Judicial Council, likely in the hundreds of millions annually. This bill could hamper the ability of trial courts and their justice partners to use several tools, including SARATSO for sex offenders, and many others, such as pretrial risk assessment tools minimize bias in pretrial detention situations. Developers of these tools, including nonprofits, universities, and even the state of Virginia, are unlikely to make any adjustments as would be required under this bill. As such, the entire judicial branch could potentially lose the use of all pretrial risk assessment tools, as their developers do not necessarily have a strong financial incentive to adhere to this bill’s stringent requirements. Courts also use tools such as MyCitation, which would also be affected. MyCitation is a tool created by the Judicial Council pursuant to AB 143 (Committee on Budget) Stats. 2021, ch. 79, designed as an easy-to-use option for defendants to provide relevant financial information for the court to consider reductions in infraction fines and fees owed.
- Unknown, potentially significant costs to the state funded trial court system (Trial Court Trust Fund, General Fund). By authorizing a new civil penalty with statutory damages, this bill will likely lead to additional case filings that otherwise would not have been commenced, with attendant workload and resource costs to the court. This bill may result in additional trial court costs to the extent that cases are brought to litigate its constitutionality. The fiscal impact of this bill to the courts will depend on many unknowns, including the number of cases filed and the factors unique to each case. An eight-hour court day costs approximately $10,500 in staff in workload. This is a conservative estimate, based on the hourly rate of court personnel including at minimum the judge, clerk, bailiff, court reporter, jury administrator, administrative staff, and jury per-diems. If court days exceed 10, costs to the trial courts could reach hundreds of thousands of dollars. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations.
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with this bill’s notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate, for all affected state agencies, costs may be in the hundreds of millions of dollars annually ongoing.
SUPPORT: (Verified 8/29/25)
SEIU California (source)
TechEquity Action (source)
ACCE Action (Alliance of Californians for Community Empowerment)
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Civil Liberties Advocacy
California Community Foundation
California Employment Lawyers Association
California Federation of Labor Unions, AFL-CIO
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, AFT, AFL-CIO
Child Care Law Center
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Housing Now!
Justice2jobs Coalition
Kapor Center
LA Defensa
LAANE (Los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Mujeres Unidas Y Activas
Northern California District Council of the International Longshore and Warehouse Union
Oakland Privacy
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Tech Oversight Project
UDW/AFSCME Local 3930
UFCW - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
OPPOSITION: (Verified 8/29/25)
Advanced Medical Technology Association
Aerospace and Defense Alliance of California
American College of Obstetricians & Gynecologists - District IX
American Innovators Network
American Property Casualty Insurance Association
American Staffing Association
America's Physician Groups
Associated General Contractors of California
Association of California Life and Health Insurance Companies
Association of National Advertisers
Business Software Alliance
CalBroadband
Calcom Association
California Association of Health Plans
California Chamber of Commerce
California Credit Union League
California Dental Association
California Fuels and Convenience Alliance
California Hospital Association
California Life Sciences
California Manufacturers and Technology Association
California Medical Association
California Radiological Society
California Staffing Professionals
Chamber of Progress
Civil Justice Association of California
College Board
Computer and Communications Industry Association
Connected Commerce Council
Consumer Data Industry Association
Consumer Technology Association
CTIA
Delta Dental of California
El Dorado County Chamber of Commerce
El Dorado Hills Chamber of Commerce
Elk Grove Chamber of Commerce
Financial Technology Association
Folsom Chamber of Commerce
Innovative Lending Platform Association
Internet Works
Kaiser Permanente
Lincoln Chamber of Commerce
Los Angeles County Business Federation
National Association of Mutual Insurance Companies
Ochin, INC.
Orange County Business Council
Personal Insurance Federation of California
Psinapse Techology
Public Risk Innovation, Solutions, and Management
Rancho Cordova Chamber of Commerce
Receivables Management Association International
Rocklin Area Chamber of Commerce
Roseville Area Chamber of Commerce
Security Industry Association
Shingle Springs/cameron Park Chamber of Commerce
Silicon Valley Leadership Group
Society for Human Resource Management
Software Information Industry Association
STAR Staffing
Sutter Health
TechCA
TechNet
United Chamber Advocacy Network UCAN
University of California
Valley Industry and Commerce Association
Yuba Sutter Chamber of Commerce
ARGUMENTS IN SUPPORT: A broad coalition, including the sponsors, SEIU California and TechEquity Action, write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile,[1] to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones.[2] Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives.
ARGUMENTS IN OPPOSITION: A coalition of financial industry groups, including the California Credit Union League, writes:
While Section 22756.2(b)(1) provides the authority to a financial institution to deny an opt-out request for financial services providers regulated under the Gramm Leach Bliley Act, the nation’s most robust law governing how financial information is used, it is unclear to us why there is not a similar exemption from the entirety of the bill. As written today, financial institutions could choose to categorically deny any consumer’s opt-out request. This will ultimately frustrate and confuse consumers, which is the opposite of the state intention of AB 1018 – which is to bring more transparency.
The Equal Credit Opportunities Act (ECOA) and Fair Credit Reporting Act (FCRA) already require creditors to provide a notice to consumers when they are denied credit detailing the reason for the denial and a method to dispute the accuracy or completeness of any information that led to the denial. Yet, financial services must still provide an opportunity to correct information, which is duplicative of FCRA, and an opportunity to appeal, which can impact the lending process. Establishing a consumer appeal provision would require a type of manual underwriting that has not been regularly used in the mortgage industry for nearly three decades. The current mortgage origination process relies on automated systems such as AUSs and Credit Scoring Models (CSMs) that do not have usable or scalable alternative manual processes.
ASSEMBLY FLOOR: 50-16, 6/2/25
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Ávila Farías, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Caloza, Connolly, Elhawary, Fong, Gabriel, Garcia, Gipson, Mark González, Haney, Harabedian, Hart, Irwin, Jackson, Kalra, Krell, Lee, Lowenthal, McKinnor, Muratsuchi, Nguyen, Ortega, Papan, Patel, Pellerin, Quirk-Silva, Celeste Rodriguez, Rogers, Blanca Rubio, Schiavo, Schultz, Sharp-Collins, Solache, Stefani, Ward, Wicks, Wilson, Zbur, Rivas
NOES: Castillo, Chen, Davies, DeMaio, Dixon, Ellis, Gallagher, Jeff Gonzalez, Hadwick, Hoover, Lackey, Macedo, Patterson, Sanchez, Ta, Tangipa
NO VOTE RECORDED: Ahrens, Alanis, Bains, Carrillo, Flora, Pacheco, Petrie-Norris, Ramos, Ransom, Michelle Rodriguez, Soria, Valencia, Wallis
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
8/30/25 12:36:06
**** END ****
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2025 - 2026 Regular Session
AB 1018 (Bauer-Kahan) - Automated decision systems
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
AB 1018 regulates the development and deployment of automated decision systems.
Fiscal Impact:
- Unknown, potentially significant costs (local funds, General Fund) to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the hundreds of millions of dollars annually statewide. For example, counties will be required to review all of their systems for compliance—including tools used for online testing, hiring, permitting, case management, risk assessments, and property management. They will also incur costs of audits, training, staffing, disclosures, appeals process, data retention, sending out notices, and potential legal exposure. If developers raise their cost as a result of this bill, these costs will be passed onto the counties. Counties may claim reimbursement of those costs if the Commission on State Mandates determines that this bill creates a new program or imposes a higher level of service on local agencies.
- The State Controller’s Office (SCO) reports that it, and all other state governmental entities would be considered a “deployer,” of an ADS, as defined in the bill. If SCO implemented one or multiple ADS systems that meet the criteria for oversight in AB 1018, SCO would anticipate needing the following General Fund Resources:
- 2025-26: $1,408,000 for 7.0 two-year limited-term positions and one-time costs;
- 2026-27: $1,384,000 for 7.0 two-year limited-term positions; and,
- 2027-28 and ongoing: $761,000 for 4.0 permanent positions.
- 2025-26: $1,408,000 for 7.0 two-year limited-term positions and one-time costs;
SCO indicates that these estimates reflect the requirements for a large, complex agency such as the SCO managing sensitive personal and financial data across automated systems. Actual staffing may be adjusted depending on the complexity and scale of the deployed ADS. Some functions may overlap or be handled by existing staff with appropriate upskilling, while external contractors could augment capacity for short-term surges, such as during initial implementation or major audits. If AB 1018 becomes law, the SCO would need 5–7 information security staff—including a dedicated supervisor—for the initial implementation of AB 1018, with 3–4 (including the supervisor) required for ongoing security and compliance operations related to automated decision systems.
- Costs (special fund) of an unknown but potentially significant amount to the Labor Commissioner to enforce Labor Code violations, possibly in the millions of dollars, annually. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- The Department of Justice (DOJ) estimates a fiscal impact of $2.5 million or less (General Fund, Unfair Competition Law Fund). The DOJ notes that implementation of this bill will be dependent upon the appropriation of funds. The DOJ will be unable to absorb the costs to comply with or implement the requirements of the bill within existing budgeted resources. The Consumer Protection Section (CPS), Civil Rights Enforcement Section (CRES) and Healthcare Rights & Access Section (HRA) within the Public Rights Division of the DOJ anticipate increased workloads in enforcing this bill beginning on January 1, 2026, and ongoing. The workload includes investigating and prosecuting violations. The Sections will require the following additional resources:
- CPS: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution;
- CRES: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution; and,
- HRA: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution.
- CPS: 1.0 Deputy Attorney General (DAG); 1.0 Associate Governmental Program Analyst (AGPA); 1.0 Legal Secretary (LS); and, $250,000 for External Consultant for expert witness services to support investigation and prosecution;
- The Civil Rights Department (CRD) estimates a fiscal impact over $2 million (General Fund). CRD notes that this bill places requirements on ADS developers and deployers. It also provides the CRD with the authority to bring a civil action to enforce compliance with this bill. CRD anticipates requesting eight positions, including researchers/ research data specialists, associate governmental program analysts, and attorneys, approximately $2.2 million for FY 26-27, and $2.1 million for FY 27-28 and ongoing. In addition to covering the costs of the new positions, these resources would also fund an ongoing contract with an expert consultant, IT and other administrative costs, and a one-time job analysis that would be required for one of the research positions. These resources would be necessary to support the complex legal and technological analysis required to enforce the components of this bill.
- The State Water Board estimates a significant fiscal impact (General Fund, special funds) likely in the millions of dollars per year. The State Water Board reports, as written, this bill’s definition of automated decision system is vague, ambiguous, and could encompass many current tools used, like excel workbooks. These tools are used broadly across Water Boards programs, and many are used to inform actions that could be considered consequential actions under the bill. To meet the bill’s provision, the State Water Board estimates significant cost pressures, likely in the millions of dollars per year. Costs would allow the State Water Board to fulfill several requirements regarding each of these covered systems. This would include facilitating and conducting a performance evaluation, contracting with an independent third party to perform audits on any systems the Water Boards develops, notifying those impacted by the consequential decision about the use of the ADS, and providing an appeal process. Taking these steps for each covered system, if feasible at all, would require significant additional staff and contract resources in the millions of dollars annually, impacting all Water Boards’ fund sources. Alternatively, State Water Board programs would need to find ways to store, process, and analyze data to make decisions without using any tool which would qualify as an automated decision system. Costs would be incurred from all eligible State Water Board fund sources.
- UC Health estimates a fiscal impact of $42 million in one-time costs and approximately $24 million in ongoing costs to implement AB 1018. Compliance costs for the University’s health systems include expenses related to implementing required audits, administrative processes, staff training, and system modifications.
- The California Department of Health and Human Services (CalHHS) reports a significant fiscal impact (General Fund, special funds). The Office of Technology and Solutions Integration (OTSI) estimates costs, across the 12 CalHHS departments, in the low millions up to low tens of millions (Total Fund) annually as solutions applicable under the bill are expected to grow in use over the coming years. These costs are estimated and attributable due to legal and security resources for monitoring and enforcement, compliance oversight, IT development and staffing necessary for governance and appeals management. This bill could impact many of the Agency's information systems and data solutions as Generative AI solutions are being built into many of the tools used. Evaluation and governance will become ongoing cost-pressure as it becomes necessary to review not just risk/compliance related to AI components of solutions, but applicability to the requirements of this bill. This bill would require programming and processing revisions, stakeholder engagement, development and operation of an appeal process in all cases, along with a third-party audit (beginning in 2030). The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the appeals processes that are yet to be determined. This estimate does not include potential increased contracting costs due to vendors including costs for conducting the assessment and other requirements.
- The California Department of Corrections and Rehabilitation (CDCR) reports that it does not currently use platforms it considers to be Automated Decision Systems (ADS). As a result, CDCR does not anticipate immediate costs associated with the requirements in AB 1018. However, should the department choose to implement or develop systems in the future that meet the bill’s definition of an ADS, there could be costs associated with compliance. These may include but are not limited to performance evaluations, third-party audits, disclosure and documentation requirements, and staff oversight responsibilities outlined in the bill. It should be noted that CDCR uses several risk assessment tools, including, but not limited to, the Static-99R to assess every eligible sex offender prior to release on parole, the California Static Risk Assessment tool to assess the risk of reoffending, the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) to assess an incarcerated individuals needs and target those needs, and the Automated Reentry Management System (ARMS). If this bill is interpreted to apply to these tools, CDCR, and other correctional agencies, could potentially incur a significant fiscal impact.
- The Department of Finance (DOF) reports potential unknown costs (General Fund) and indicates that that this bill may apply to its ongoing pilot using generative artificial intelligence to assist with bill analyses.
- Unknown, potentially significant costs to the Judicial Council, likely in the hundreds of millions annually. This bill could hamper the ability of trial courts and their justice partners to use several tools, including SARATSO for sex offenders, and many others, such as pretrial risk assessment tools minimize bias in pretrial detention situations. Developers of these tools, including nonprofits, universities, and even the state of Virginia, are unlikely to make any adjustments as would be required under this bill. As such, the entire judicial branch could potentially lose the use of all pretrial risk assessment tools, as their developers do not necessarily have a strong financial incentive to adhere to this bill’s stringent requirements. Courts also use tools such as MyCitation, which would also be affected. MyCitation is a tool created by the Judicial Council pursuant to AB 143 (Committee on Budget) Stats. 2021, ch. 79, designed as an easy-to-use option for defendants to provide relevant financial information for the court to consider reductions in infraction fines and fees owed.
- Unknown, potentially significant costs to the state funded trial court system (Trial Court Trust Fund, General Fund). By authorizing a new civil penalty with statutory damages, this bill will likely lead to additional case filings that otherwise would not have been commenced, with attendant workload and resource costs to the court. This bill may result in additional trial court costs to the extent that cases are brought to litigate its constitutionality. The fiscal impact of this bill to the courts will depend on many unknowns, including the number of cases filed and the factors unique to each case. An eight-hour court day costs approximately $10,500 in staff in workload. This is a conservative estimate, based on the hourly rate of court personnel including at minimum the judge, clerk, bailiff, court reporter, jury administrator, administrative staff, and jury per-diems. If court days exceed 10, costs to the trial courts could reach hundreds of thousands of dollars. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations.
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with this bill’s notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate, for all affected state agencies, costs may be in the hundreds of millions of dollars annually ongoing.
Background:
ADS, especially those powered by AI, are being increasingly deployed in a multitude of contexts, including employment, housing, education, and healthcare. Major transparency and fairness concerns have been raised about the use of ADS to make consequential decisions, essentially determinations with significant legal or other material effect on people’s lives.
According to the SEIU California and TechEquity Action, sponsors of this bill: “While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile, to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones. Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.”
Opponents of this measure include a broad scope of industries that will be impacted by its requirements. Writing in opposition, a coalition of financial industry groups, including the California Credit Union League, point out that “Financial services are already comprehensively regulated by multiple state and federal frameworks that focus on outcomes and consumer protection” and that “Current federal and state laws already explicitly prohibit bias and discrimination in financial decisions.” They contend, that the broad scope of the bill would create significant operational challenges for financial institutions of all sizes and the proposed requirements could slow innovation and limit access to financial services for underserved populations.
Writing on behalf of a broad coalition representing physicians, hospitals and health systems, pharmaceutical research, health plans, and other health care stakeholders, California Life Sciences points out that that this legislation, while well-intentioned, would harm patient care, exacerbate existing health disparities, and negatively impact the overall health care delivery system. The compliance costs associated with AB 1018 would be substantial. These costs would be passed on to payers and ultimately individual Californians, leading to higher health care expenses. In a state already grappling with high health care costs, it is crucial to avoid policies that would exacerbate this issue.
Colorado Governor Jared Polis recently called a special legislative session set to begin August 21, 2025 to address Colorado’s AI law (the Colorado Artificial Intelligence Act or “CAIA”), which is set to take effect February 1, 2026, to address federal preemption considerations, the state’s budget issues, and to potentially delay the bill’s implementation. CAIA is the nation’s first comprehensive state-level regulation of artificial intelligence systems. Similar to this bill, CAIA, applies to AI systems for consequential decisions, and requires impact assessments, disclosures, appeals and correction rights, and document keeping and record retention. CAIA faced immediate industry backlash in Colorado, and the state’s AI task force warned in February that the law’s terminology and compliance burdens could overwhelm smaller employers and create uncertainty for developers.
Proposed Law:
- Defines all of the following:
- “Artificial intelligence” as an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
- “Automated decision system” as a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons. “Automated decision system” does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
- “Consequential decision” as a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
- Employment-related decisions;
- Education and vocational training as they relate to assessment and placement, detecting student cheating and plagiarism, accreditation, certification, admissions and enrollment, discipline, evaluation, financial aid and scholarships, and proctoring;
- Housing and lodging as they relate to rental or short-term housing and lodging, home appraisals, rental subsidies, publicly supported housing, and essential utilities.
- Family planning, adoption services, reproductive services, and assessments related to child protective services.
- Health care and health insurance, including mental health care, dental, and vision;
- Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor;
- The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration;
- Legal services;
- Private arbitration;
- Mediation;
- Elections as they relate voting, redistricting, voter eligibility and registration, distribution of voting information, and election administration;
- Access to government benefits or services or assignment of penalties by a government entity;
- Places of public accommodation;
- Insurance; and,
- Internet and telecommunications access.
- “Covered automated decision system” or “covered ADS” as an automated decision system that is designed or used to make or facilitate a consequential decision.
- “Deployer” as a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
- “Developer” as a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
- “Developer-approved use” as a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision. “Developer-approved use” includes any reasonably foreseeable fine tuning of the covered ADS.
- “Disparate impact” as a differential effect on a group of individuals who share a protected characteristic.
- “Disparate treatment” as differential treatment of an individual or group of individuals on the basis of a protected characteristic.
- “Employment-related decision” as a decision made by an employer, either directly or through a third party, that affects wages, benefits, other compensation, hours, work schedule, performance evaluation, hiring, discipline, promotion, termination, job tasks and responsibilities, assignment of work, access to work and training opportunities, productivity requirements, workplace health and safety, or other terms or conditions of employment.
- “Express consent” as an affirmative written authorization that is granted in response to a notice that is clear, meaningful, and prominent, conveyed in a manner that a natural person would notice and understand it, and not contained within a more general notice, agreement, or set of terms and conditions. “Express consent” does not mean an authorization that is either inferred from inaction, or obtained through the use of a dark pattern, as defined.
- “Fine-tune” as to adjust the model parameters of an automated decision system through exposure to additional data.
- “Labor Commissioner” as Chief of the Division of Labor Standards Enforcement.
- “Substantial modification” as a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs. “Substantial modification” does not mean a modification those results from fine tuning.
- With respect to a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, the developer of the covered ADS shall conduct an initial performance evaluation on the covered ADS before January 1, 2027, and annually thereafter.
- With respect to a covered ADS that is first deployed or made available to potential deployers on or after January 1, 2026, the developer of the covered ADS shall conduct a performance evaluation on the covered ADS before initially deploying the covered ADS, or making the covered ADS available to potential deployers, and annually thereafter.
- In conducting a performance evaluation on a covered ADS, a developer shall do all of the following:
- Describe the purpose of the covered ADS.
- List and describe all developer-approved uses of the covered ADS.
- For each developer-approved use, evaluate the expected performance of the covered ADS and document all of the following:
- The expected accuracy and reliability of the covered ADS.
- Any reasonably foreseeable effects of fine tuning on the accuracy and reliability of the covered ADS.
- Whether any disparate treatment is intended to occur and, if so, all of the following:
- The conditions under which each disparate treatment is intended to occur.
- Whether each disparate treatment is necessary for a developer-approved use.
- Whether any alternatives not involving disparate treatment were considered.
- Any reasonably foreseeable effects of fine tuning on disparate treatment.
- Whether any disparate impacts are reasonably likely to occur and, if so, all of the following:
- The conditions under which each disparate impact is reasonably likely to occur.
- Whether each disparate impact is necessary for a developer-approved use.
- Whether any alternatives not involving disparate impacts were considered.
- Whether any measures have been taken by the developer to mitigate the risk of unanticipated disparate impacts resulting from the use of the covered ADS.
- With respect to a covered ADS that has been deployed, whether any unanticipated disparate impacts have been reported to the developer by a deployer, and whether the developer has taken any measures to mitigate those disparate impacts.
- Any reasonably foreseeable effects of fine tuning on disparate impacts.
- Beginning January 1, 2030, contract with an independent third-party auditor to assess the developer’s compliance.
- Provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance. A developer that provides documentation to an auditor make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information, the developer shall notify the auditor and provide a basis for the withholding.
- Describe the purpose of the covered ADS.
- If the deadline for conducting a performance evaluation elapses before the audit has been completed, a developer shall not deploy the covered ADS or make the covered ADS available to potential deployers until the audit has been completed.
- A developer that receives feedback from an auditor shall do both of the following:
- Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
- Make a high-level summary of the feedback publicly available at no cost to users of the developer’s internet website.
- Consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
- A developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer shall provide the deployer with all of the following:
- The results of the most recent performance evaluation conducted on the covered ADS.
- For each developer-approved use of the covered ADS, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
- For each developer-approved use of the covered ADS, a description of whether and under what circumstances the covered ADS can be fine-tuned.
- An explanation of the deployer’s responsibilities under this chapter, including a description of any circumstances under which the deployer would assume the responsibilities of a developer.
- Any technical information necessary for the deployer to comply with this chapter. A developer shall not be required to provide additional technical information to a deployer that has assumed the responsibilities of a developer.
- A developer that provides documentation to a potential deployer pursuant to this subdivision may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a developer withholds information the developer shall notify the deployer and provide a basis for the withholding.
- The results of the most recent performance evaluation conducted on the covered ADS.
- A developer that receives an impact assessment from an auditor of a deployed covered ADS provide all of the following information to any deployer of the covered ADS:
- Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
- Any material differences between the expected reliability of the covered ADS and the observed reliability of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
- Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
- An explanation of any steps the deployer can take to mitigate these discrepancies.
- Any material differences between the expected accuracy of the covered ADS and the observed accuracy of the covered ADS and the deployment conditions under which those differences are reasonably likely to occur.
- A developer that receives feedback from an auditor shall consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS.
- A developer that provides documentation to a deployer pursuant to this section shall ensure the documentation is all of the following:
- Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
- Provided in English and in any other language the developer regularly uses to communicate with deployers.
- Presented in a manner that ensures the communication clearly and effectively conveys the required information to the deployer.
- Transmitted directly to the deployer or otherwise made available in a manner reasonably calculated to ensure the deployer receives the documentation.
- A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall maintain all of the following documentation in an unredacted format for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers plus 10 years:
- The results of any performance evaluations conducted on the covered ADS.
- Any documentation provided to deployers.
- Any documentation provided to, or received from, auditors.
- Records of any redactions.
- The results of any performance evaluations conducted on the covered ADS.
- It is unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
- A developer that deploys a covered ADS or makes a covered ADS available to potential deployers shall designate at least one employee to oversee the developer’s compliance with this chapter. A developer shall require the designated employee to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
- This section does not apply to a developer subject to the federal Gramm-Leach-Bliley Act (15 U.S.C. Sec. 6801 et seq.) who uses a covered ADS to make or facilitate a consequential decision related to financial services, as described.
- Unless the subject of the consequential decision is having a medical emergency, before a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information:
- A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
- The name, version number, and developer of the covered ADS.
- Whether the deployer’s use of the covered ADS is within the scope of a developer-approved use and a description of that use.
- The personal characteristics or attributes of the subject that the covered ADS measures or assesses to make or facilitate the consequential decision.
- The sources of personal information collected from the subject to make or facilitate the consequential decision.
- (Any key parameters known to disproportionately affect the outcome of the consequential decision.
- The structure and format of the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
- Whether a natural person will review either of the following before the consequential decision is finalized:
- The outputs of the covered ADS.
- The outcome of the consequential decision.
- The outputs of the covered ADS.
- The subject’s rights and the means and timeframe for exercising those rights.
- Contact information for the deployer.
- Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
- Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
- A statement informing the subject that a covered ADS will be used to make or facilitate the consequential decision.
- Unless the subject of the consequential decision is having a medical emergency, after a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with a plain language written disclosure containing all of the following information within five days:
- The personal characteristics or attributes of the subject that the covered ADS used to make or facilitate the consequential decision.
- The sources of personal information collected from the subject that were used to make or facilitate the consequential decision.
- Any key parameters that disproportionately affected the outcome of the consequential decision.
- The structure and format of the outputs of the covered ADS and a plain language description of how those outputs were used to make or facilitate the consequential decision.
- The role that the ADS played in making the consequential decision and whether any human judgment was involved.
- Contact information for the deployer.
- Contact information for the entity that manages the covered ADS, if that entity is not the deployer.
- Contact information for the entity that will interpret the results of the covered ADS, if that entity is not the deployer.
- The subject’s rights and the means and timeframe for exercising those rights.
- The personal characteristics or attributes of the subject that the covered ADS used to make or facilitate the consequential decision.
- A deployer that provides documentation to a subject of a consequential decision shall ensure the documentation is all of the following:
- Transmitted directly to the subject.
- Provided in English and in any other language that the deployer regularly uses to communicate with subjects.
- Made available in formats that are accessible to people who are blind or have other disabilities.
- Presented in a manner that ensures the communication clearly and effectively conveys the required information to the subject.
- Transmitted directly to the subject.
- After a deployer finalizes a consequential decision made or facilitated by a covered ADS, the deployer shall provide any subject of that decision with an opportunity to do both of the following within 30 business days:
- Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision. A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request. If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision. If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
- A deployer that denies a request to correct personal information shall do both of the following:
- Provide the subject with an explanation of the basis for the denial.
- Provide the subject with a reasonable opportunity to request that the deployer delete the subject’s personal information.
- Appeal the outcome of the consequential decision.
- Correct any incorrect personal information used by the covered ADS to make or facilitate the consequential decision. A deployer shall comply with a request to correct personal information within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request. If a deployer determines that complying with a request to correct personal information would change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, rectify the decision. If a deployer determines that complying with a request to correct personal information would not change the outcome of the consequential decision, the deployer shall, within 30 days of making the determination, inform the subject that the correction was made but that it did not alter the decision.
- A deployer shall review a request to appeal a consequential decision within 30 business days of receiving the request if the request is accompanied by documentation sufficient to assess the basis for the request. If a deployer determines that the original decision was incorrect, the deployer shall, within 30 days of making the determination, rectify the decision. If a deployer determines that the original decision was correct, the deployer shall, within 30 days of making the determination, inform the subject that the consequential decision will not be altered.
- A deployer that denies a request to appeal the outcome of a consequential decision shall provide the subject with an explanation of the basis for the denial.
- A deployer that provides documentation to the subject of a consequential decision may make reasonable redactions for the purpose of protecting trade secrets. To the extent that a deployer withholds information, the deployer shall notify the subject and provide a basis for the withholding.
- A deployer’s collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
- A deployer that does any of the following assumes the responsibilities of a developer:
- Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS during the three-year period.
- Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer’s use of the covered ADS is outside the scope of a developer-approved use.
- Substantially modifies an automated decision system and does either of the following:
- Uses the substantially modified system to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period.
- Makes the substantially modified system available to potential deployers.
- Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period, if the deployer did not receive any documentation from the developer of the covered ADS during the three-year period.
- A deployer that uses a covered ADS to make or facilitate a consequential decision shall retain the following documentation in an unredacted format for as long as the deployer uses the covered ADS plus 10 years:
- Any documentation received from developers pursuant to this chapter.
- Any documentation provided pursuant to this section to subjects of consequential decisions made or facilitated by the covered ADS.
- Any requests to correct personal information.
- Any requests to appeal the outcome of a consequential decision.
- Any documentation provided to, or received from, auditors.
- Records of any redactions made.
- Any documentation received from developers pursuant to this chapter.
- A deployer that uses a covered ADS to make or facilitate a consequential decision shall designate at least one employee to oversee the deployer’s compliance with this chapter. A deployer shall require the designated employee to conduct a prompt and comprehensive review of any credible compliance issue related to the deployer’s use of a covered ADS that is raised to that employee.
- Requires, within 30 days of receiving a request from the Attorney General for a performance evaluation, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the performance evaluation to the Attorney General.
- Allows the Attorney General may share performance evaluations with other enforcement entities as necessary for enforcement purposes.
- Provides that the disclosure or sharing of a performance evaluation does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained in the performance evaluation.
- Provides that a performance evaluation disclosed or shared is exempt from the California Public Records Act.
- Makes each day a covered ADS is used for which a performance evaluation has not been submitted to the Attorney General an additional violation.
- Authorizes any of the following public entities may bring a civil action against a developer, deployer, or auditor for a violation for injunctive relief, declaratory relief, reasonable attorney’s fees and litigation costs and a civil penalty of up to $25,000 per violation:
- The Attorney General;
- A district attorney, county counsel of any county within which a city has a population in excess of 750,000, city attorney in a city and county, or city attorney of a city having a population in excess of 750,000;
- A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney;
- The Civil Rights Department; and,
- The Labor Commissioner with respect to employment-related decisions only.
- The Attorney General;
- Provides that a developer or deployer who contracts with a third party to perform the developer’s or deployer’s duties under this chapter, other than those duties related to auditing, is subject to liability for the third party’s failure to perform those duties.
- Provides that these provisions do not apply to an automated decision system the sole purpose of which is to do either of the following:
- Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems; or,
- Operate aircraft in the national airspace.
- Detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems; or,
- Provides that the use of a consumer credit score to inform a consequential decision does not itself create an obligation under this chapter.
- Provides that, in an action alleging the defendant’s development, modification, or use of an automated decision system is alleged to have caused or facilitated the violation, the extent to which the defendant complied with these provisions is relevant to, but not conclusive of, whether the defendant committed a violation.
Related Legislation:
AB 331 (Bauer-Kahan), of the 2023-24 Legislative Session, would have required deployers and developers to conduct annual impact assessments of the potential impacts of their ADS and prohibited use of ADS that contributed to discrimination. AB 331 was held on the Assembly Appropriations Committee’s suspense file.
AB 2930 (Bauer-Kahan), of the 2023-24 Legislative Session, would have implemented a regulatory framework for use of ADS by government and private entities, with the goal of to preventing algorithmic discrimination. Amendments made in this Committee limited the bill’s scope to private deployers of ADS and consequential decisions relating to employment. AB 2930 died on the Senate Inactive File.
This bill is one of a many related to AI this Legislative Session:
- SB 53 (Weiner) establishes a consortium develop a framework for the creation of a public cloud computing cluster to advance the development of AI that is safe, ethical, equitable, and sustainable. SB 53 is pending in the Assembly Appropriations Committee.
- SB 366 (Smallwood Cuevas) creates a study evaluating the impact of AI on worker well-being. SB 366 was held under submission in this Committee.
- SB 420 (Padilla) regulates high-risk automated decision systems. SB 420 was not head in the Assembly Committee on Privacy and Consumer Protection.
- SB 468 (Becker) imposes a duty on business that deploy a high-risk AI systems that processes personal information to protect personal information. SB 468 was held under submission in this Committee.
- SB 503 (Weber Pierson) requires developers of patient care decision support tools and health facilities to make reasonable efforts to identify uses of patient care decision support tools in health programs. SB 503 is pending in the Assembly Appropriations Committee.
- SB 524 (Arreguin) requires law enforcement agencies to note when they use AI on official reports. SB 524 is pending in the Assembly Appropriations Committee.
- SB 579 (Padilla) establishes a mental health and AI working group. SB 579 was held under submission in this Committee.
- SB 833 (McNerney) requires a state agency in charge of critical infrastructure that deploys AI to establish a human oversight mechanism. SB 833 is pending in the Assembly Appropriations Committee.
- AB 222 (Bauer-Kahan) requires reporting about energy use related to AI. AB 222 is pending in this Committee.
- AB 316 (Krell) prohibits a defendant that used AI from asserting a defense that the AI autonomously caused the harm to the plaintiff. AB 316 is pending on this Committee’s Suspense File.
- AB 410 (Wilson) requires bots using AI to disclose that they are bots. AB 410 is pending in this Committee.
- AB 412 (Bauer Kahan) requires a of a generative AI model to document any copyrighted materials used to train the model. AB 412 was not heard in the Senate Judiciary committee at the request of the author.
- AB 489 (Bonta) makes provisions of law that prohibit the use of specified terms, letters, or phrases to falsely indicate or imply possession of a license or certificate to practice a health care profession enforceable against an entity who uses AI. AB 489 is pending in this Committee.
- AB 853 (Wicks) requires a large online platform to retain any available provenance data in content posted on the large online platform. AB 853 is pending in this Committee.
- AB 979 (Irwin) develops a California AI Cybersecurity Collaboration Playbook to facilitate information sharing across the AI community. AB 979 is pending in this Committee.
- AB 1064 (Bauer-Kahan) adopts criteria for determining the level of estimated risk of an AI system on children. AB 1064 is pending in this Committee.
- AB 1159 (Addis) prohibits using student personal information to train AI. AB 1159 is was not heard at the request of the author in the Assembly Committee on Privacy and Consumer Protection.
- AB 1405 (Bauer-Kahan) establishes a mechanism allowing natural persons to report misconduct by AI auditors. AB 1405 is pending in this Committee.
-- END --
Senator Thomas Umberg, Chair
2025-2026 Regular Session
Bill No: AB 1018 (Author:Bauer-Kahan)
Version: July 3, 2025
Hearing Date: July 15, 2025
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
Automated decision systems
DIGEST
This bill regulates the use of “automated decision systems (ADS).” It places obligations on developers and deployers of such systems designed or used to make or facilitate “consequential decisions.”
EXECUTIVE SUMMARY
ADS, especially those powered by AI, are being increasingly deployed in a multitude of contexts, including employment, housing, education, and health care. Major transparency and fairness concerns have been raised about the use of ADS to make consequential decisions, essentially determinations with significant legal or other material effect on people’s lives.
This bill seeks to regulate the development and deployment, by both public and private actors, of “covered ADS,” systems that make consequential decisions, or decisions that materially impact the cost, terms, quality, or accessibility of specified services, opportunities, access, resources, and treatment for natural persons. It requires developers to conduct performance evaluations of their ADS, which, among other things, identify details of the systems, expected performance and uses, and potential disparate impacts. Deployers are required to provide certain notices to subjects of consequential decisions and afford certain rights to them, including the right to opt out, correct information, and appeal final decisions.
This bill is sponsored by TechEquity Action and the Service Employees International Union (SEIU) California. It is supported by a variety of labor, consumer, privacy, and other advocacy groups, including Common Sense Media and Consumer Reports. It is opposed by a number of industry associations, including the Consumer Technology Association and the American Staffing Association.
PROPOSED CHANGES TO THE LAW
Existing law:
- Establishes the CCPA, which grants consumers certain rights with regard to their personal information, including enhanced notice, access, and disclosure; the right to deletion; the right to restrict the sale of information; and protection from discrimination for exercising these rights. It places attendant obligations on businesses to respect those rights. (Civ. Code § 1798.100 et seq.)
- Establishes the Consumer Privacy Rights Act (CPRA), which amends the CCPA and creates the California Privacy Protection Agency (PPA), which is charged with implementing these privacy laws, promulgating regulations, and carrying out enforcement actions. (Civ. Code § 1798.100 et seq.; Proposition 24 (2020).)
- Requires the Attorney General to adopt regulations governing access and opt-out rights with respect to businesses’ use of automated decisionmaking technology, including profiling and requiring businesses’ response to access requests to include meaningful information about the logic involved in those decisionmaking processes, as well as a description of the likely outcome of the process with respect to the consumer. (Civ. Code § 1798.185(a)(16).)
- Provides that beginning the later of July 1, 2021, or six months after the PPA provides notice to the Attorney General that it is prepared to begin rulemaking, the authority assigned to the Attorney General to adopt regulations under this section shall be exercised by the PPA. (Civ. Code § 1798.185(d).)
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act (FEHA). (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that meets the following criteria:
- Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use.
- Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civ. Code § 3426.1(d).)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms:
- “ADS” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. “Automated decision system” does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
- “High-risk automated decision system” means an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. (Gov. Code § 11546.45.5.)
This bill:
- Requires a developer of a covered ADS that was first deployed, or made available to potential deployers, before January 1, 2026, to conduct an initial performance evaluation on the covered ADS before January 1, 2027. For covered ADS first deployed or made available on or after January 1, 2026, the developer shall conduct a performance evaluation before initially deploying it or making it available to potential deployers.
- Requires developers to additionally conduct a performance evaluation on the covered ADS under any of the following circumstances:
- Following any substantial modification by the developer or any fine tuning that materially changes the uses or outputs of the covered ADS.
- No more than one year after the developer last conducted a performance evaluation, for as long as the developer deploys the covered ADS or makes the covered ADS available to potential deployers.
- Requires a developer, in conducting a performance evaluation, to do the following:
- Describe the purpose of the covered ADS.
- List and describe all developer-approved uses of the covered ADS.
- For each developer-approved use, evaluate the expected performance of the covered ADS and document various metrics, including the expected accuracy and reliability of the ADS, whether any disparate treatment is intended to occur, and whether any disparate impacts are reasonably likely to occur.
- Contract with an independent third-party auditor to assess the developer’s compliance herewith.
- Requires a developer to consider and attempt to incorporate any feedback from an auditor into the development of any subsequent version of the covered ADS and to make a high-level summary of the feedback publicly available.
- Requires a developer that sells, licenses, or otherwise transfers a covered ADS to a potential deployer to provide the deployer with all of the following:
- The results of the most recent performance evaluation.
- For each developer-approved use, instructions explaining how the covered ADS should be used by the deployer to make or facilitate a consequential decision.
- For each developer-approved use, a description of whether and under what circumstances the covered ADS can be fine-tuned.
- An explanation of the deployer’s responsibilities hereunder, as specified.
- Any technical information necessary for the deployer to comply with this chapter.
- Permits a developer to make reasonable redactions from the documentation provided to protect trade secrets, but requires notice of such withholding and the basis for it.
- Requires a developer that receives an impact assessment from an auditor of a deployed covered ADS to provide all of the following information to any deployer of the covered ADS:
- Any material differences between the expected accuracy or reliability and the observed accuracy or reliability and the deployment conditions under which those differences are reasonably likely to occur.
- Any unanticipated disparate impacts resulting from the use of the covered ADS and the deployment conditions under which those disparate impacts are reasonably likely to occur.
- An explanation of any steps the deployer can take to mitigate these discrepancies.
- Dictates the process and form that any documentation required to be provided to a deployer by a developer must take.
- Makes it unlawful to advertise to consumers in the state that a covered ADS is capable of performing in a manner not substantiated by the results of the most recent performance evaluation conducted on the covered ADS.
- Requires developers to designate at least one employee to oversee the developer’s compliance herewith and to conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
- Requires a deployer, before it finalizes a consequential decision made or facilitated by a covered ADS, to provide any subject of that decision with a plain language written disclosure containing specified information, including:
- Notice that a covered ADS will be used.
- Details identifying the covered ADS.
- Whether the deployer’s use of the covered ADS is within the scope of a developer-approved use and a description of that use.
- The personal characteristics or attributes of the subject used by the ADS to make the decision.
- The sources of personal information collected from the subject to make or facilitate the consequential decision.
- Details regarding the outputs of the covered ADS and a plain language description of how those outputs are used to make or facilitate the consequential decision.
- Whether a natural person will review the outputs of the covered ADS or the outcome of the consequential decision.
- The subject’s rights pursuant hereto.
- Contact information for the deployer and the entity or entities managing the ADS and interpreting the results.
- Requires a deployer, before finalizing a consequential decision, to provide any subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS, except as specified. A deployer that denies a request to opt out shall provide the subject with an explanation of the basis for the denial.
- Requires a deployer, after finalizing a consequential decision, to provide any subject of that decision with a plain language written disclosure containing specified information within five days, including most of the same information described above that is provided before finalizing the decision. The deployer must also provide the subject with an opportunity to correct any incorrect personal information used and to appeal the decision according to specified processes.
- Dictates the process and form that documentation required to be provided to a subject by a deployer must take.
- Provides that a deployer’s collection, use, retention, and sharing of personal information from a subject of a consequential decision shall be reasonably necessary and proportionate to achieve the purposes for which the personal information was collected and processed, or for another disclosed purpose that is compatible with the context in which the personal information was collected, and not further processed in a manner that is incompatible with those purposes.
- Requires a deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting more than 6,000 people in a given three-year period to contract with an independent third-party auditor to conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter.
- Prescribes the process an auditor must take when conducting an impact assessment of a deployer’s covered ADS, including the information that must be requested and the details that must be documented. After completing the assessment, the auditor is required to provide the results to the deployer and to make a high-level summary of the assessment publicly available, as provided.
- Provides the conditions under which a deployer assumes the responsibilities of a developer, including when the deployer:
- Uses a covered ADS to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period and certain conditions are met.
- Substantially modifies an ADS and uses it to make or facilitate consequential decisions that directly impact more than 6,000 people in a given three-year period or makes it available to potential deployers.
- Requires a deployer that uses a covered ADS to make or facilitate a consequential decision to designate at least one employee to oversee the deployer’s compliance herewith. That employee must conduct a prompt and comprehensive review of any credible compliance issue related to the deployer’s use of a covered ADS that is raised to that employee.
- Requires a developer or deployer to provide, where applicable, an auditor with any available information that is reasonably necessary for the auditor to conduct a comprehensive audit.
- Provides that necessary audits not completed by the specified deadlines require either the developer to not deploy or make available the ADS or the deployer to not use the ADS to make consequential decisions.
- Imposes retention requirements on developers and deployers with respect to specified documentation.
- Includes exceptions to specified provisions in the case of a medical emergency.
- Provides that the provisions governing deployers become operative on January 1, 2027.
- Allows for the redaction of information revealing trade secrets, as provided, but requires disclosure of such withholding and the basis for it.
- Provides that within 30 days of receiving a request from the Attorney General (AG) for a performance evaluation or impact assessment, a developer, deployer, or auditor of a covered ADS shall provide an unredacted copy of the document to the AG. The AG is authorized to share these materials with other enforcement entities as necessary for enforcement purposes.
- Clarifies that the above disclosure or sharing of a performance evaluation or impact assessment does not constitute a waiver of any attorney-client privilege, work-product protection, or trade secret protection that might otherwise exist with respect to any information contained therein. Such materials are exempt from the California Public Records Act.
- Authorizes enforcement actions against developers, deployers, or auditors to be brought by the following entities:
- The Attorney General.
- A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
- A city prosecutor in any city having a full-time city prosecutor with the consent of the district attorney.
- The Civil Rights Department.
- The Labor Commissioner with respect to employment-related decisions only.
- Authorizes a court to award a prevailing plaintiff in such actions the following:
- Injunctive relief.
- Declaratory relief.
- Reasonable attorney’s fees and litigation costs.
- A civil penalty of up to $25,000 per violation.
- Exempts ADS whose sole purpose is to detect, protect against, or respond to cybersecurity incidents or preserve the integrity or security of computer systems or to operate aircraft in the national airspace. The use of a consumer credit score to inform a consequential decision does not itself create an obligation hereunder.
- Clarifies the relevance of compliance, or the failure to comply, with these provisions in actions pursuant to the Unruh Civil Rights Act and the Fair Employment and Housing Act.
- Defines AI, ADS, trade secret, and personal information in accordance with existing law and defines other relevant terms, including:
- “Consequential decision” means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
- Employment-related decisions.
- Education and vocational training, as specified.
- Housing and lodging, as specified.
- Essential utilities, as specified.
- Family planning, adoption services, reproductive services, and assessments related to child protective services.
- Health care and health insurance, including mental health care, dental, and vision.
- Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
- The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
- Legal services.
- Private arbitration.
- Mediation.
- Elections, as specified.
- Access to government benefits or services or assignment of government penalties.
- Places of public accommodation, as defined.
- Insurance.
- Internet and telecommunications access.
- “Covered ADS” means an ADS that is designed or used to make or facilitate a consequential decision.
- “Substantial modification” means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs.
- “Consequential decision” means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
COMMENTS
- Considerations for development and deployment of ADS
With recent dramatic advances in the capabilities of AI systems, the need for regulatory frameworks for accountability and responsible development and deployment have become ever more urgent. This is especially true with respect to AI-powered ADS that are used to make, or assist in making, decisions that have a legal or other significant effect.
ADS introduce several concerning issues when deployed across various sectors. Bias and discrimination represent perhaps the most significant problem, as AI systems frequently reflect and amplify historical biases present in their training data. This can lead to unfair outcomes based on protected characteristics like race, gender, and socioeconomic status, particularly in sensitive domains such as hiring, lending, housing allocation, and criminal justice.
The lack of transparency in many AI systems compounds these concerns. These technologies often function as “black boxes” where the rationale behind specific decisions remains obscure even to their developers. This opacity makes it exceptionally difficult for affected individuals to understand why they were denied a loan, were passed over for a job opportunity, or received an unfavorable outcome. Such obscurity directly challenges meaningful accountability when harmful outcomes inevitably occur.
Accuracy and reliability issues also persist even in sophisticated AI systems. These technologies can make confident but incorrect predictions, with errors often disproportionately affecting already marginalized groups. Performance demonstrated in controlled testing environments frequently fails to translate to complex real-world scenarios, leading to unexpected and harmful outcomes.
Accountability gaps emerge when determining responsibility for AI-caused harms. The complex relationship between developers, deployers, and users makes liability difficult to establish. Legal frameworks consistently lag behind rapidly advancing technological capabilities, creating environments where harms can occur without clear recourse.
By reducing complex human situations to algorithmic outputs, ADS risk eliminating human judgment, empathy, and contextual understanding from important processes. Many people report feeling powerless when facing decisions made by automated systems, especially when those systems lack transparency or meaningful appeal mechanisms.
For instance, Amazon’s deployment of ADS for hiring purposes provides an example of how bias can be built into these systems:
Amazon.com Inc’s machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants’ resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters.
Automation has been key to Amazon’s e-commerce dominance, be it inside warehouses or driving pricing decisions. The company’s experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars - much like shoppers rate products on Amazon, some of the people said.
“Everyone wanted this holy grail,” one of the people said. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.[1]
An additional example involved a discrimination charge by the Department of Housing and Urban Development against Meta, which emphasized the importance of knowing which characteristics are being considered by an ADS.[2] The settled claims involved Meta targeting users with “housing ads based on algorithms that relied partly on characteristics protected under the Fair Housing Act, like race, national origin and sex.” The charges also alleged that “Meta’s lookalike or special ad audience tool allowed advertisers to target users based on protected traits.”
In response to growing concerns about the increased deployment of ever-advanced ADS, the Biden Administration published a Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the design, use, and deployment of AI to protect the rights of the American public:
- Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
- Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
. . .
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.[3]
- Ensuring accountability and transparency in ADS development and deployment
This bill seeks to implement some of the principles laid out in the blueprint discussed above in an effort to regulate “covered ADS,” defined to mean an ADS that is designed or used to make or facilitate a “consequential decision.” “Consequential decision” means a decision that materially impacts the cost, terms, quality, or accessibility of any of a series of things to a natural person. This includes educational and employment opportunities, housing, health care, as well as reproductive, financial, legal, and government services.
- Developers
First, the bill places a series of obligations on a person or entity that designs, codes, substantially modifies, or otherwise produces an ADS that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes. These developers are required to conduct a performance evaluation on covered ADS. These evaluations describe the purpose and approved uses of the systems and evaluate the expected performance of it, including the expected accuracy and reliability.
To proactively address potential adverse outcomes, the evaluations have to document whether any disparate treatment is intended to occur and whether any disparate impacts are reasonably likely to occur, and indicate measures taken to mitigate the risk of unintended impacts.
For ADS first deployed before January 1, 2026, an initial performance evaluation must be conducted before January 1, 2027. If first deployed after that date, the evaluation must occur before initially deploying the ADS or making it available to potential deployers. Thereafter, performance evaluations are to be conducted at least annually as well as following any substantial modification or specified fine tuning of the ADS.
Developers are required to provide potential deployers with the results of these performance evaluations along with additional information about how to deploy the systems and the intended uses.
A developer that deploys a covered ADS or makes a covered ADS available to potential deployers must also designate at least one employee to oversee the developer’s compliance herewith and that employee must conduct a prompt and comprehensive review of any credible compliance issue raised to that employee.
- Deployers
This bill also places obligations on a person or entity that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
- Pre- and post-use notice and disclosure
Before it finalizes a consequential decision made or facilitated by a covered ADS, a deployer must provide any subject of that decision with notice that an ADS will be used and a plain language written disclosure containing a detailed set of information. This includes identifying details of the ADS used, whether the deployer’s use is developer-approved, and the types and sources of information processed by the system.
Deployers must disclose the structure and format of the outputs of the covered ADS and a description of how those outputs are used to make or facilitate the consequential decision. A subject must be told whether a natural person will review the outputs of the covered ADS or the outcome of the consequential decision before the relevant decision is finalized.
After a deployer finalizes a consequential decision, the deployer must again provide any subject of that decision with a plain language written disclosure within five days identifying the role the covered ADS played and containing much of the same information as above.
- Other rights: opt out, correction, and appeal
Before finalizing a consequential decision made or facilitated by a covered ADS, the deployer is required to provide the subject of that decision with a reasonable opportunity to opt out of the use of the covered ADS. A deployer may deny a request to opt out in certain limited circumstances, such as where the deployer is subject to the federal Gramm-Leach-Bliley Act, as provided, or the subject of the consequential decision is having a medical emergency. An explanation of the basis for the denial must then be provided.
Once a consequential decision has been made by a covered ADS, the deployer must provide the subject with an opportunity to correct any incorrect personal information used and to appeal the decision within 30 business days. Requests to correct or appeal must be reviewed, as specified, and determinations must be communicated, as provided.
- The use of independent audits for proper AI oversight
The bill also calls for independent audits. The use of auditing can be a key tool in effectively assessing how automated and AI-powered systems are working and what their impacts are. Audits can ensure legal compliance and, when shared publicly, afford a measure of transparency. Mandatory audits create baseline standards across the industry, making it easier to evaluate systems and ensure minimum safety thresholds. This levels the playing field and prevents a “race to the bottom” where competitive pressures lead companies to skimp on safety and quality-control measures. Audit requirements create transparency that builds public confidence in these systems, especially those used in critical domains like healthcare, criminal justice, or financial services. When people know systems have been independently verified, they are more likely to accept and appropriately use them.
This accountability also provides recourse when things go wrong, as qualified auditors can provide concrete evidence for regulatory decisions and legal proceedings. They create a paper trail showing whether companies exercised reasonable care, which is crucial for determining liability when systems cause harm. Ultimately, they afford a measure of assurance that any legal guidelines are complied with, which is especially critical given the limited resources and technical expertise of state agencies.
Last year, the National Telecommunications and Information Administration (NTIA) published an “Artificial Intelligence Accountability Policy Report.” One of its main recommendations focused on the utility of such independent auditing:
Independent AI audits and evaluations are central to any accountability structure. To help create clarity and utility around independent audits, we recommend that the government work with stakeholders to create basic guidelines for what an audit covers and how it is conducted – guidance that will undoubtedly have some general components and some domain-specific ones. This work would likely include the creation of auditor certifications and audit methodologies, as well as mechanisms for regulatory recognition of appropriate certifications and methodologies.
Auditors should adhere to consensus standards and audit criteria where possible, recognizing that some will be specific to particular risks (e.g., dangerous capabilities in a foundation model) and/or particular deployment contexts (e.g., discriminatory impact in hiring). Much work is required to create those standards – which NIST and others are undertaking. Audits and other evaluations are being rolled out now concurrently with the development of technical standards. Especially where evaluators are not yet relying on consensus standards, it is important that they show their work so that they too are subject to evaluation. Auditors should disclose methodological choices and auditor independence criteria, with the goal of standardizing such methods and criteria as appropriate. The goals of safeguarding sensitive information and ensuring auditor independence and appropriate expertise may militate towards a certification process for qualified auditors.
AI audits should, at a minimum, be able to evaluate claims made about an AI system’s fitness for purpose, performance, processes, and controls.[4]
This bill requires auditing for developers and, in limited circumstances, for deployers. Developers are required to contract with an independent third-party auditor to assess the developer’s compliance with bill’s requirements. A developer that receives feedback from an auditor must consider and attempt to incorporate that feedback into the development of any subsequent version of a covered ADS and to share certain details with potential deployers. Developers must also make a high-level summary of the feedback publicly available at no cost to users of the developer’s internet website.
A deployer that uses a covered ADS to make or facilitate consequential decisions directly impacting 6,000 or more people in a given three-year period is also required to contract with an independent third-party auditor. The auditor must conduct an impact assessment on the covered ADS before January 1, 2030, and every three years thereafter, as applicable. The auditor is to provide the results to the deployer and also make a high-level summary of the results available to the public.
Developers and deployers are required to provide the auditor with any available information that is reasonably necessary for the auditor to comprehensively assess developer compliance or to conduct a comprehensive impact assessment of the covered ADS, respectively. Reasonable redactions are permitted to protect trade secrets, but the developer or deployer must indicate the withholding and the basis for it.
- Enforcement
Specified public prosecutors are authorized to bring civil actions against developers, deployers, or auditors who violate the provisions of the bill. This includes the AG, district attorneys, city attorneys, county counsel, the Civil Rights Department, and the Labor Commissioner, as provided. A court can award a prevailing plaintiff injunctive or declaratory relief, reasonable attorney’s fees and litigation costs, and a civil penalty of up to $25,000 per violation.
The bill requires developers, deployers, and auditors to provide an unredacted copy of a performance evaluation or impact assessment to the AG within 30 days of a request. The AG is authorized to share with other enforcement entities, but only as necessary for enforcement purposes. Each day a covered ADS is used for which a performance evaluation or impact assessment has not been submitted to the Attorney General after 30 days of a request is an additional violation.
The bill also amends part of the Unruh Civil Rights Act and FEHA to clarify the relevance of compliance with this bill and liability under those laws.
- Other attempts at regulating ADS
A number of jurisdictions have stepped forward to respond to the dramatic increase in ADS usage. For instance, the European Union AI Act provides guardrails for what it deems “high-risk AI systems” to ensure transparency and fairness. Here in the United States, a number of states have introduced legislation in this space, including New York and Connecticut. However, the first comprehensive state-level regulation has come in Colorado.
The Colorado law, approved by their Governor on May 17, 2024, places requirements on developers and deployers to use reasonable care to protect consumers from the risks of algorithmic discrimination.
On February 20, 2025, the Virginia Legislature passed the High-Risk Artificial Intelligence Developer and Deployer Act (Virginia AI Act), a comprehensive artificial intelligence bill focused on preventing algorithmic discrimination. However, on March 24, 2025, Virginia Governor Glenn Youngkin vetoed it.
In addition, SB 420 (Padilla, 2025) seeks to regulate the development and deployment of “high-risk ADS,” as defined, requiring impact assessments and maintenance of governance programs.
SB 7 (McNerney, 2025) also regulates the use of ADS in the employment context, providing a minimum of 30 days notice to workers before deployment as well as post-deployment rights to notice, to correct information, and to appeal. It also restricts certain uses of ADS in the employment context.
All of these bills, including this one, have harmonized definitions for what an ADS is, based off of the definition in existing law that makes clear that an ADS is one that is used to “assist or replace human discretionary decisionmaking and materially impacts natural persons.”
- Stakeholder positions
According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities
The sponsors of the bill, SEIU California and TechEquity Action, along with a broad coalition of groups, including Smart Justice California and UFCW Western States Council, write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile,[5] to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones.[6] Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives. Specifically, the legislation:
- Requires these tools to be tested before they are used on the public: Requires that people who make and use these tools test them to ensure they do not create harm and comply with our existing rights to non-discrimination before they are sold and used on the public. It ensures that these tests are verified by an independent third party.
- Provides a notice to people that this tool will be used to make a critical decision about their life: Provides people the information they need to understand where these tools are showing up in their lives and how they’ll be used to determine their housing, healthcare, and job outcomes.
- Provides an explanation to people who were subject to a decision made with these tools: Provides people an explanation of what the tool did, what personal information it used about them to make the decision and what role the tool played in making the decision.
- Ensures that everyday people have more control over how these tools are used in their daily lives: Through this bill, people will have the right to opt out of the use of an ADS tool in a critical decision about them; they will be able to correct information that the tool used to make the decision if it is inaccurate; and they will have the right to appeal the decision.
A coalition of financial industry groups, including the California Credit Union League, write in opposition:
While Section 22756.2(b)(1) provides the authority to a financial institution to deny an opt-out request for financial services providers regulated under the Gramm Leach Bliley Act, the nation’s most robust law governing how financial information is used, it is unclear to us why there is not a similar exemption from the entirety of the bill. As written today, financial institutions could choose to categorically deny any consumer’s opt-out request. This will ultimately frustrate and confuse consumers, which is the opposite of the state intention of AB 1018 – which is to bring more transparency.
The Equal Credit Opportunities Act (ECOA) and Fair Credit Reporting Act (FCRA) already require creditors to provide a notice to consumers when they are denied credit detailing the reason for the denial and a method to dispute the accuracy or completeness of any information that led to the denial. Yet, financial services must still provide an opportunity to correct information, which is duplicative of FCRA, and an opportunity to appeal, which can impact the lending process. Establishing a consumer appeal provision would require a type of manual underwriting that has not been regularly used in the mortgage industry for nearly three decades. The current mortgage origination process relies on automated systems such as AUSs and Credit Scoring Models (CSMs) that do not have usable or scalable alternative manual processes.
In response to these concerns, the author has agreed to an amendment that specifies the following: “A developer subject to the federal Gramm-Leach-Bliley Act who uses a covered ADS to make or facilitate a consequential decision related to financial services, as described in paragraph (7) of subdivision (c) of Section 22756, is exempt from this section, 22756.1.”
Writing in support, the California Professional Firefighters argue:
AI and algorithms are being used to make decisions in areas ranging from healthcare to elections, shaping our lives in ways that could not have been imagined even a decade previous. While the term “intelligence” in the name implies a level of sophistication and mechanical impartiality for the programs, time and again it has been demonstrated that these pieces of software come loaded with the biases of their makers, eliminating the nuance that is inherent in the human condition to make overly-generalized decisions that can have disastrous consequences.
AB 1018 would enact significant oversight for these decision-making programs, ensuring that individuals are aware of how AI is impacting their job, health, housing, or other crucial aspects of their life. Additionally, this measure would put in place auditing tools to review the programs and assess if they are functioning as intended and making accurate, non-discriminatory decisions. It is crucial that this new technology is only deployed in manners that benefit humanity, not streamline acts of discrimination or oppression.
A large coalition of industry associations, led by the California Chamber of Commerce, writes in opposition:
We appreciate the absence of a private right of action. As you know, because compliance is not easy to achieve in areas where massive changes in public policy are sought and where the state of law and technology are not only complicated but constantly evolving, a private right of action would have been highly problematic and chilling of innovation. That being said, businesses are still subject to civil enforcement by not only the Attorney General (AG) but by all other public attorneys (city prosecutors, district attorneys, city attorneys, and county counsel), in addition to the administrative enforcement of the Civil Rights Department (CRD), and the Labor Commissioner with respect to employment-related decisions.
In response to concerns about the breadth of enforcement, the author has agreed to an amendment that narrows the local public prosecutors that can enforce the bill’s provisions.
The Chamber coalition also argues:
Untenable opt-out and pre-and post-decision notice obligations must be deleted. All opt out and notice obligations, including the right to appeal which will have drastically different impact depending on context, must be deleted in full to make the bill workable. Such requirements are not only largely unworkable in many contexts, but also wholly unrelated to and unnecessary for there to be a bill that would require evaluations/assessments that to help reduce bias and discriminatory outcomes from the development and deployment of such ADS.
In response to these concerns, the author has agreed to amendments that completely remove the right of subjects of ADS decisions to opt out, as well as rework the pre-use notice provision to only require a generalized notice, not one personalized for each subject.
Additionally, the Chamber’s coalition in opposition asserts:
Third-party auditor requirements should be deleted in favor of self-assessments, particularly when third-party auditing would effectively grant a monopoly today and creates a costly cottage industry tomorrow, with access to highly sensitive and proprietary information, and no standards of care or liability. The third-party auditor requirement imposes excessive and unnecessary costs on businesses, increasing IP risks and operational inefficiencies, without providing any added consumer protections. In fact, the requirement will have the opposite effect by driving up costs for consumers. With a limited number of auditors available, a legal mandate would create a surge in demand, allowing existing auditors to charge inflated fees without competition. As businesses absorb these new compliance costs, they will be forced to raise prices, ultimately burdening consumers, reducing sales, and hindering economic growth in California. Furthermore, this requirement exceeds the scope of other U.S. laws and proposals, making California the most expensive jurisdiction for compliance. Given such impacts, the third-party auditor requirement for both developers and deployers must be removed to make this bill remotely viable.
In response to these concerns, the author has agreed to a series of major amendments. First, the auditing requirement, as applied to developers, will be delayed until 2030 to allow the relevant market to take shape. Second, the requirement for developers to conduct performance evaluations will be limited to no more than once a year. Finally, the author has agreed to completely remove the auditing requirements for deployers to reduce the burden on deployers, despite the benefits of having this layer of oversight.
SUPPORT
SEIU California (sponsor)
TechEquity Action (sponsor)
ACCE Action (Alliance of Californians for Community Empowerment)
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Civil Liberties Advocacy
California Community Foundation
California Employment Lawyers Association
California Federation of Labor Unions, AFL-CIO
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, AFT, AFL-CIO
Child Care Law Center
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center (EPIC)
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Housing Now!
Justice2jobs Coalition
Kapor Center
LA Defensa
LAANE (Los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Mujeres Unidas Y Activas
Northern California District Council of the International Longshore and Warehouse Union (ILWU)
Oakland Privacy
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Tech Oversight Project
UDW/AFSCME Local 3930
UFCW - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
OPPOSITION
Advanced Medical Technology Association (ADVAMED)
Aerospace and Defense Alliance of California
American College of Obstetricians & Gynecologists - District IX
American Innovators Network
American Property Casualty Insurance Association
American Staffing Association
America's Physician Groups
Associated General Contractors of California
Association of California Life and Health Insurance Companies
Association of National Advertisers
Business Software Alliance
CalBroadband
Calcom Association
California Association of Health Plans
California Chamber of Commerce
California Credit Union League
California Dental Association
California Fuels and Convenience Alliance
California Hospital Association
California Life Sciences
California Manufacturers and Technology Association
California Medical Association (CMA)
California Radiological Society
California Staffing Professionals (CSP)
Chamber of Progress
Civil Justice Association of California (CJAC)
College Board
Computer and Communications Industry Association
Connected Commerce Council
Consumer Data Industry Association
Consumer Technology Association
CTIA
Delta Dental of California
El Dorado County Chamber of Commerce
El Dorado Hills Chamber of Commerce
Elk Grove Chamber of Commerce
Financial Technology Association
Folsom Chamber of Commerce
Innovative Lending Platform Association
Internet Works
Kaiser Permanente
Lincoln Chamber of Commerce
Los Angeles County Business Federation (BIZFED)
National Association of Mutual Insurance Companies
Ochin, INC.
Orange County Business Council
Personal Insurance Federation of California
Psinapse Techology
Public Risk Innovation, Solutions, and Management (PRISM)
Rancho Cordova Chamber of Commerce
Receivables Management Association International
Rocklin Area Chamber of Commerce
Roseville Area Chamber of Commerce
Security Industry Association
Shingle Springs/cameron Park Chamber of Commerce
Silicon Valley Leadership Group
Society for Human Resource Management
Software Information Industry Association
STAR Staffing
Sutter Health
TechCA
TechNet
United Chamber Advocacy Network UCAN
University of California
Valley Industry and Commerce Association (VICA)
Yuba Sutter Chamber of Commerce
RELATED LEGISLATION
Pending Legislation:
SB 7 (McNerney, 2025) See Comment 3. SB 7 is currently in the Assembly Privacy and Consumer Protection Committee.
SB 420 (Padilla, 2025) See Comment 3. SB 420 is currently in the Assembly Privacy and Consumer Protection Committee.
SB 468 (Becker, 2025) imposes a duty on a business that deploys a high-risk artificial intelligence system, or high-risk ADS, that processes personal information to protect that information and requires such a deployer to maintain a comprehensive information security program that meets specified requirements. SB 468 is currently in the Senate Appropriations Committee.
Prior Legislation:
SB 892 (Padilla, 2024) would have required CDT to develop and adopt regulations to create an ADS procurement standard, as specified, and prohibited a state agency from procuring ADS, entering into a contract for ADS, or any service that utilizes ADS, until CDT has adopted regulations creating an ADS procurement standard, as specified. SB 892 was vetoed by Governor Newsom, who stated in his veto message that aspects of the bill would disrupt ongoing work, “including existing information technology modernization efforts, which would lead to implementation delays and higher expenses for critical projects.”
AB 2885 (Bauer-Kahan & Umberg, Ch. 843, Stats. 2024) established a uniform definition for “artificial intelligence” in California’s code, which is used in this bill.
AB 2930 (Bauer-Kahan, 2024) See Comment 3. AB 2930 died without a vote on the Senate Floor.
AB 302 (Ward, Ch. 800, Stats. 2023) required CDT, on or before September 1, 2024, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
AB 331 (Bauer-Kahan, 2023) was substantially similar to AB 2930. AB 331 died in the Assembly Appropriations Committee.
PRIOR VOTES:
Assembly Floor (Ayes 50, Noes 16)
Assembly Appropriations Committee (Ayes 10, Noes 3)
Assembly Judiciary Committee (Ayes 8, Noes 3)
Assembly Privacy and Consumer Protection Committee (Ayes 9, Noes 3)
**************
Jeffrey Dastin, Insight - Amazon scraps secret AI recruiting tool that showed bias against women (October 10, 2018) Reuters, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/. All internet citations are current as of June 21, 2025. ↑
Lauren Feiner, DOJ settles lawsuit with Facebook over allegedly discriminatory housing advertising (June 21, 2022) CNBC, https://www.cnbc.com/2022/06/21/doj-settles-with-facebook-over-allegedly-discriminatory-housing-ads.html. ↑
Blueprint For An AI Bill Of Rights (October 2022) Office of Science and Technology Policy, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. ↑
Artificial Intelligence Accountability Policy Report (March 27, 2024) NTIA, https://www.ntia.gov/sites/default/files/publications/ntia-ai-report-final.pdf. ↑
https://themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms. ↑
https://www.theguardian.com/society/2019/oct/25/healthcare-algorithm-racial-biases-optum. ↑
ASSEMBLY THIRD READING
ATR Bill Id:AB 1018
Author:(Bauer-Kahan)
As Amended Ver:May 01, 2025
Majority vote
SUMMARY
This bill creates a comprehensive regime designed to ensure human oversight over automated decision systems (ADS) that are used in "consequential decisions" – those that materially impact an individual's rights, opportunities, or access to critical resources or services – in order to mitigate bias and unreliability in these systems.
Major Provisions
- Requires a developer to, among other things:
- Conduct and document initial performance evaluations on the ADS, and conduct and document subsequent performance evaluations on the ADS, no more than a year apart, as specified.
- Contract with an independent third-party auditor to assess the developer's compliance with requirements for performance evaluations.
- Make specified disclosures to deployers, including the results of the most recent performance evaluation.
- Beginning January 1, 2027, requires a deployer to, among other things:
- Before finalizing a consequential decision, provide the subjects of the consequential decision with an informational notice and an opportunity to opt out of the use of the ADS.
- After finalizing a consequential decision, provide the subjects of the decision an informational notice and an opportunity to correct a subject's incorrect personal information and appeal the outcome.
- Minimize collection, use, retention, and sharing of personal information from subjects of consequential decisions.
- If the ADS impacts more than 5,999 people in a three-year period, contract with a third-party independent auditor to impact assessments on the covered ADS, as specified. In specified circumstances, the deployer of such systems must assume the responsibilities of a developer.
- Authorizes public prosecutors, the Civil Rights Department, and the Labor Commissioner to bring civil actions to enforce violations of the bill's requirements.
COMMENTS
ADS typically use artificial intelligence (AI) that produce simplified outputs – such as scores, classifications, or recommendations – to assist or replace human discretionary decisionmaking. When used in consequential contexts – such as healthcare, employment, and criminal justice – ADS can meaningfully impact the public's rights, opportunities, or access to critical resources or services. ADS can process enormous datasets, identify hidden patterns, and make decisions with efficiency and scale that vastly exceeds human capabilities. But relying on ADS to make consequential decisions can be hazardous if the systems are not trained carefully or tested thoroughly: the datasets they are trained on are often unrepresentative or contaminated with bias, the inferences they draw from those datasets are often inscrutable, and these systems can fail to accurately account for the complexity of human behavior. Without human-centered oversight, the impacts of defective ADS can be devastating.
In 2022, the White House Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights, which identifies five principles that should "guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence."[1] These principles would (1) protect the public from unsafe or ineffective ADS, (2) prevent algorithmic discrimination, (3) protect data privacy, (4) give subjects notice and explanation when ADS are used, and (5) provide for human alternatives, consideration, and fallback. The Blueprint calls for rigorous testing, monitoring, and independent evaluation of ADS to ensure that ADS are at least as fair and trustworthy as the humans they are displacing.
This bill seeks to codify that vision. It would create a comprehensive transparency regime for developers and deployers of ADS that are used for consequential decisionmaking in order to prevent algorithmic discrimination on the basis of protected characteristics and to help ensure that ADS can be trusted to perform reliably and accurately. The bill would require deployers of ADS to give subjects of consequential decisions notice, an ability to opt out of the use of the ADS, a chance to correct any personal information used by the ADS to make the decision, and a right to appeal the outcome of the decision. The bill also provides for independent evaluation by third-party auditors. The Attorney General, local public prosecutors, Civil Rights Department, or Labor Commissioner, as applicable, may bring civil enforcement actions against violators.
According to the Author
The use of automated decision systems (ADS) has become prevalent among Californian's daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
Arguments in Support
A broad coalition of civil society and labor organizations jointly write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile, to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones. Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives. Specifically, the legislation:
- Requires these tools to be tested before they are used on the public: Requires that people who make and use these tools test them to make sure they do not create harm and comply with our existing rights to non-discrimination before they are sold and used on the public. It ensures that these tests are verified by an independent third party.
- Provides a notice to people that this tool will be used to make a critical decision about their life: Provides people the information they need to understand where these tools are showing up in their lives and how they'll be used to determine their housing, healthcare, and job outcomes.
- Provides an explanation to people who were subject to a decision made with these tools: Provides people an explanation of what the tool did, what personal information it used about them to make the decision and what role the tool played in making the decision.
- Ensures that every day people have more control over how these tools are used in their daily lives: Through this bill, people will have the right to opt out of the use of an ADS tool in a critical decision about them; they will be able to correct information that the tool used to make the decision if it is inaccurate; and they will have the right to appeal the decision.
Pre-deployment testing protects Californians by ensuring that businesses understand the risks and accuracy of the systems they're purchasing and deploying to make consequential decisions about Californians. The strong notice and explanation provisions in this bill are a critical first step for consumers to know how these tools are being used and to be able to exercise their right to correct their data or appeal a decision. These protections are more important than ever as we see a Federal pullback from safety guardrails on AI and as other states and jurisdictions pass laws recognizing the danger inherent in letting these systems enter into our lives without clear protections and safeguards for the public.
Arguments in Opposition
A coalition of industry trade organizations, led by California Chamber of Commerce, writes that the bill:
. . .fails to focus on the stated objective of requirements on high-risk automated decision systems (ADS) to the detriment of every industry in the state of California, and from small business to large. Instead, the bill broadly targets businesses of all sizes, across every industry, and regulates even lower risk applications of ADS, including those that are already in use, and unnecessarily alters what constitutes discrimination under California law. Such overreach not only exposes smaller businesses to significant – if not devastating – liability even for mere errors that caused no harm to consumers, but it would also hinder many beneficial uses of ADS, including but not limited to: enabling faster approvals and expanded access to credit; enhancing real-time fraud detection; fostering job creation and new industries; improving efficiency to help level the playing field between small and large businesses; addressing major societal challenges such as bias and discrimination, economic inequality, climate change, injustices in the criminal system, disaster relief, and humanitarian aid; and advancing new treatments for previously incurable diseases.
To be clear, whether decisions are being made by humans from start to end, or a byproduct of using or incorporating new technologies in the decision-making process, we take our responsibility to not discriminate with the utmost seriousness. We believe that algorithmic discrimination, or discrimination that results from AI-enabled technologies, is already prohibited under our anti-discrimination laws because our laws are rights-based and not technology-specific. We also agree that companies need to take care to reduce bias and discrimination in decisions that have legal impact on the provision or denial of fundamental rights or essential opportunities and services, which is why so many of them already conduct impact assessments.
But it would be incredibly short-sighted for regulation to stifle innovation when alternative (human-decision driven) systems may be equally, if not more, flawed, and when properly developed and deployed ADS can enhance fairness and accountability. While ADS may pose unique challenges in terms of bias, they also pose unique advantages to combatting it as well: for example, ADS decision-making processes can be more transparent and traceable than the exercise of human discretion. The structured nature of these tools offer opportunities for detection and correction that can be more challenging to come by in human decision-making processes where human bias is often more subtle and harder to detect. Unfortunately, we believe AB 1018 will have an undesired chilling effect on the technology and make it that much harder to develop the very tools that can help combat bias in decisionmaking.
As introduced, AB 1018 goes far beyond ensuring developers and deployers of these technologies act responsibly to adhere to existing anti-discrimination protections and veers into an indirect restriction upon the usage of the technology itself, discouraging technological innovation and usage by making it so onerous and risky that businesses are realistically pushed back toward the alternative, human driven process – which we know based on historical evidence will not help eliminate bias and discrimination in any way.
FISCAL COMMENTS
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with the bill's notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate for all affected state agencies, costs may be in the tens of millions to hundreds of millions of dollars annually ongoing. By way of illustration, the California Health and Human Services Agency (HHSA) alone anticipates compliance costs in the low millions to tens of millions of dollars annually for its 12 departments, with additional ongoing cost pressures if use of ADS tools continues to grow in the future. HHSA notes this cost estimate does not include potential increased contracting costs from vendors passing on costs for conducting assessments and other requirements in the bill.
- Likely non-reimbursable costs to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the tens of millions to hundreds of millions of dollars annually statewide.
- Costs (General Fund) to the Department of Justice (DOJ) of an unknown but significant amount, possibly in the millions of dollars annually, to receive and review ADS performance evaluations and bring civil enforcement actions. At the time this analysis was prepared, DOJ was unable to provide a cost estimate for its duties under the bill. The bill's author submitted a budget request seeking $8 million General Fund annually ongoing to fund DOJ's implementation of the bill.
- Costs (General Fund) to CRD to enforce compliance with the bill's requirements. CRD anticipates annual costs around $2.1 million to support the "complex legal and technological analysis required to enforce the components of this bill."
- Costs (special fund) of an unknown but potentially significant amount to the LC to enforce Labor Code violations of the bill's requirements. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- Cost pressures (Trial Court Trust Fund, General Fund) of an unknown but potentially significant amount to the courts to adjudicate enforcement actions authorized. Actual costs will depend on the number of cases filed and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2024-25 state budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
VOTES
ASM PRIVACY AND CONSUMER PROTECTION: 9-3-3
YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, McKinnor, Ortega, Pellerin, Ward, Wicks
NO: Dixon, DeMaio, Macedo
ABS, ABST OR NV: Patterson, Petrie-Norris, Wilson
ASM JUDICIARY: 8-3-1
YES: Kalra, Bauer-Kahan, Bryan, Connolly, Harabedian, Papan, Stefani, Zbur
NO: Dixon, Macedo, Sanchez
ABS, ABST OR NV: Pacheco
ASM APPROPRIATIONS: 10-3-2
YES: Wicks, Calderon, Caloza, Elhawary, Fong, Mark González, Hart, Pacheco, Pellerin, Solache
NO: Dixon, Ta, Tangipa
ABS, ABST OR NV: Sanchez, Arambula
UPDATED
VERSION: May 01, 2025
CONSULTANT: Josh Tosney / P. & C.P. / (916) 319-2200 FN: 0000917
The White House, "Blueprint for an AI Bill of Rights," (Oct. 2022), p. 14, https://bidenwhitehouse.archives.gov/ostp/ai-bill-of-rights/. Despite the use of the term "AI" in its title, the Blueprint focuses on ADS. ↑
Date of Hearing: May 14, 2025
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
ABPCA Bill Id:AB 1018
Author:(Bauer-Kahan) – As Amended Ver:May 1, 2025
Policy Committee: | Privacy and Consumer Protection | Vote: | 9 - 3 |
Judiciary | 8 - 3 | ||
|
|
Urgency: No State Mandated Local Program: No Reimbursable: No
SUMMARY:
This bill establishes requirements for the developer and deployer of an automated decision system (ADS) used to make or facilitate a consequential decision.
Under the bill, “deployer” means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, directly or through a third-party contract.
“Consequential decision” includes decisions that materially impact the cost, terms, quality, or accessibility of specified subjects, including employment, education and vocational training, essential utilities, health care, financial services, and legal services, among others.
Specifically, among other provisions, this bill:
- Requires a developer to, among other things:
- Conduct and document initial performance evaluations on the ADS, and conduct and document subsequent performance evaluations on the ADS, no more than a year apart, including in specified circumstances.
- Contract with an independent third-party auditor to assess the developer’s compliance with requirements for performance evaluations.
- Make specified disclosures to deployers, including the results of the most recent performance evaluation.
- Beginning January 1, 2027, requires a deployer to, among other things:
- Before finalizing a consequential decision, provide the subjects of the consequential decision with an informational notice and an opportunity to opt out of the use of the ADS to make or facilitate the consequential decision.
- After finalizing a consequential decision, provide the subjects of the decision an informational notice and an opportunity to correct a subject’s incorrect personal information or appeal the outcome.
- Minimize collection, use, retention, and sharing of personal information from subjects of consequential decisions.
- If the ADS impacts more than 5,999 people in a three-year period (“high-impact ADS”), contract with a third-party independent auditor to conduct regular impact assessments on the covered ADS, as specified. In specified circumstances, the deployer of a high-impact ADS must assume the responsibilities of a developer.
- Comply with any additional requirements related of covered ADS promulgated by the California Privacy Protection Agency in duly adopted regulations.
- Establishes requirements for auditors conducting impact assessments for deployers’ high-impact ADS.
- Authorizes public prosecutors, the Civil Rights Department (CRD), and Labor Commissioner (LC) to bring specified civil actions to enforce violations of the bill’s requirements.
- Provides that in an action alleging a violation of the Unruh Civil Rights Act or Fair Employment and Housing Act in which an ADS is alleged to have committed or facilitated the violation, the defendant’s compliance with this bill is relevant to, but not conclusive of, the claim.
FISCAL EFFECT:
- Costs (General Fund, special funds) of an unknown but significant amount to each state agency that deploys ADS to make or contribute to consequential decisions and must comply with the bill’s notice, opt-out, correction, appeal, and, if applicable, audit requirements. In the aggregate for all affected state agencies, costs may be in the tens of millions to hundreds of millions of dollars annually ongoing. By way of illustration, the California Health and Human Services Agency (HHSA) alone anticipates compliance costs in the low millions to tens of millions of dollars annually for its 12 departments, with additional ongoing cost pressures if use of ADS tools continues to grow in the future. HHSA notes this cost estimate does not include potential increased contracting costs from vendors passing on costs for conducting assessments and other requirements in the bill.
- Likely non-reimbursable costs to local agencies that deploy ADS to make consequential decisions and must comply with the above-described requirements, possibly in the tens of millions to hundreds of millions of dollars annually statewide.
- Costs (General Fund) to the Department of Justice (DOJ) of an unknown but significant amount, possibly in the millions of dollars annually, to receive and review ADS performance evaluations and bring civil enforcement actions. At the time this analysis was prepared, DOJ was unable to provide a cost estimate for its duties under the bill. The bill’s author submitted a budget request seeking $8 million General Fund annually ongoing to fund DOJ’s implementation of the bill.
- Costs (General Fund) to CRD to enforce compliance with the bill’s requirements. CRD anticipates annual costs around $2.1 million to support the “complex legal and technological analysis required to enforce the components of this bill.”
- Costs (special fund) of an unknown but potentially significant amount to the LC to enforce Labor Code violations of the bill’s requirements. Actual costs will depend on the number of enforcement actions and the amount of workload associated with each action.
- Cost pressures (Trial Court Trust Fund, General Fund) of an unknown but potentially significant amount to the courts to adjudicate enforcement actions authorized. Actual costs will depend on the number of cases filed and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The fiscal year 2024-25 state budget provides $37.3 million ongoing General Fund to backfill declining revenue to the Trial Court Trust Fund.
COMMENTS:
- Purpose. According to the author:
[T]he algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes…Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted.
- Background. ADS, and artificial intelligence more generally, are widely used in increasingly sophisticated ways to help companies and government agencies make decisions. For example, employers use ADS to screen resumes for job openings, lenders use ADS to recommend prospective homeowners for mortgages, and hospital risk-scoring systems use ADS to predict care needs for patients. However, as detailed in the policy committee analyses of this bill, research shows these systems can reproduce historical patterns of discrimination, stereotype, and bias.
This bill establishes a state regulatory framework of ADS that imposes testing and disclosure requirements on developers and deployers of the technology, and authorizes enforcement by DOJ, the LC, and CRD for violations. The bill requires deployers of the technology, including state and local government entities, to conduct initial and ongoing performance assessments of each ADS, provide people affected by covered ADS notice and opportunities to opt out or correct their information in the system, and, for high-impact ADS, contract with third-party auditors.
- Related Legislation. AB 1405 (Bauer-Kahan) requires the Government Operations Agency to create and administer a program that allows an AI auditor to enroll with the agency and allows a member of the public to report misconduct by an enrolled AI auditor. AB 1405 is pending on this committee’s suspense file.
- Prior Legislation. AB 331 (Bauer-Kahan), of the 2023-24 Legislative Session, would have required deployers and developers to conduct annual impact assessments of the potential impacts of their ADS and prohibited use of ADS that contributed to discrimination. AB 331 was held on this committee’s suspense file.
AB 2930 (Bauer-Kahan), of the 2023-24 Legislative Session, would have implemented a regulatory framework for use of ADS by government and private entities, with the goal of to preventing algorithmic discrimination. Amendments made in the Senate Appropriations Committee limited the bill’s scope to private deployers of ADS and consequential decisions relating to employment. AB 2930 died on the Senate Inactive File.
Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081
Date of Hearing: April 29, 2025
ASSEMBLY COMMITTEE ON JUDICIARY
Ash Kalra, Chair
ABPCA Bill Id:AB 1018
Author:(Bauer-Kahan) – As Amended Ver:April 10, 2025
As Proposed to be Amended
SUBJECT: Automated decision systems
KEY ISSUE: Should the Legislature enact a comprehensive framework regulating automated decision systems (ADS) that materially affect Californians’ access to essential services and opportunities by requiring third-party audits; establishing consumer rights; and incorporating compliance obligations into California’s civil rights law?
SYNOPSIS
AB 1018 responds to the growing use of automated decision systems (ADS) in high-stakes sectors such as employment, housing, health care, and credit, where machine-driven tools are increasingly used to assist or replace human decisionmaking. While these systems offer efficiency and scale, numerous studies and real-world examples have shown that they can also replicate or exacerbate discriminatory outcomes—often in ways that are opaque to affected individuals and difficult to remedy after the fact. This bill represents the third and most comprehensive iteration of the author’s effort to create a regulatory framework around such systems. Building on two prior measures by the same author—AB 331 (2023) and AB 2930 (2024)—AB 1018 synthesizes and extends both approaches by establishing clear obligations for both developers and deployers of covered ADS. It requires performance evaluations, third-party audits for high-volume systems, plain-language notice to individuals, and opportunities to opt out, correct data, or appeal a decision. It also imposes recordkeeping requirements and mandates that documentation be shared with the Attorney General upon request. Importantly, AB 1018 does not define or create a new standard of unlawful discrimination. Rather, it provides that compliance with the bill’s obligations—such as performing an audit, providing notice, or mitigating known harms—may be considered in determining whether a violation of existing civil rights law has occurred, but is not dispositive. Specifically, the bill amends Civil Code Section 51 (Unruh Civil Rights Act) to state that a developer’s or deployer’s compliance with these requirements is “relevant to, but not conclusive of” a finding of unlawful discrimination. This preserves the existing standard for establishing intentional discrimination under the Unruh Act while recognizing that procedural diligence may weigh in either direction when evaluating liability.
As proposed to be amended, the bill clarifies that the definitions of “disparate treatment” and “disparate impact” in the bill are intended solely to guide internal compliance, evaluation, and documentation requirements under the Business and Professions Code, and are not intended to modify the legal standards or burdens of proof applied by courts in actions under the Unruh Civil Rights Act. Supporters of the bill—comprising a broad coalition of civil rights, labor, and consumer organizations—argue that the bill imposes common-sense safeguards to ensure that ADS tools are tested before being used in critical contexts, and that people are given basic rights to understand, challenge, or opt out of opaque algorithmic decisions. Opponents, led by the California Chamber of Commerce and joined by various industry associations, argue that the bill is overly broad in scope, imposes impractical notice and audit requirements, and risks chilling innovation—particularly among small businesses. This measure was heard in the Assembly’s Committee on Privacy and Consumer Protection and passed by a vote of 9-3. The proposed amendments are reflected in the SUMMARY below, and throughout the analysis.
SUMMARY: Establishes a regulatory framework for high-risk automated decision systems by requiring performance evaluations, third-party audits, consumer transparency rights, and enforcement by designated state and local agencies. Specifically, this bill:
- Defines key terms, including:
- “Automated decision system,” or ADS, means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons. The term does not include a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
- “Consequential decision” means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
- Employment-related decisions, as defined.
- Education and vocational training as they relate to specified categories.
- Housing and lodging as they relate to rentals or short-term housing and lodging, home appraisals, rental subsidies, and publicly supported housing.
- Essential utilities: electricity, heat, water, transportation, and municipal trash and sewage services.
- Family planning, adoption services, reproductive services, and assessments related to child protective services.
- Health care and health insurance.
- Financial services.
- The criminal justice system with respect to pretrial release, sentencing, and alternatives to incarceration.
- Legal services, private arbitrations, and mediation.
- Elections, as they relate to specified categories.
- Access to government benefits or services or assignment of penalties by a government entity.
- Places of public accommodation, as specified.
- Insurance.
- Internet and telecommunications access.
- “Covered ADS” means an ADS that is designed or used to make or facilitate a consequential decision.
- “Deployer” means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
- “Developer” means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
- “Developer-approved use” means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision, and includes any reasonably foreseeable fine tuning of the covered ADS.
- “Disparate impact” means a differential effect on a group of individuals who share a protected characteristic.
- “Disparate treatment” means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
- “Express consent” means affirmative written authorization granted in response to a notice that meets specified requirements.
- “Fine-tune” means to adjust the model parameters of an ADS through exposure to additional data.
- “Personal information” has the same meaning as defined in the California Consumer Privacy Act.
- “Protected characteristic” means a characteristic listed in the Unruh Civil Rights Act.
- “Substantial modification” means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs, but does not include modifications resulting from fine-tuning.
- “Trade secret” has the same meaning as provided under the Uniform Trade Secrets Act.
- Requires developers to:
- Conduct initial performance evaluations on ADS either before they are first deployed or, if they were deployed before January 1, 2026, by January 1, 2027.
- Conduct subsequent performance evaluations on ADS no more than a year apart, including following any substantial modification or fine-tuning.
- In conducting a performance evaluation, for each developer-approved use, document:
- The expected accuracy and reliability of the covered ADS.
- Whether any disparate treatment is intended to occur, whether such treatment is necessary, and whether alternatives were considered.
- Whether the covered ADS is reasonably likely to result in disparate impacts, whether these impacts are necessary, and whether alternatives were considered.
- Whether measures were taken to mitigate the risk of unanticipated disparate impacts, including any unanticipated disparate impacts reported by deployers.
- Any reasonably foreseeable effects of fine tuning, as specified.
- Contract with an independent third-party auditor to assess the developer’s compliance with requirements for performance evaluations. Provide the auditor with necessary documentation, which may be redacted to protect trade secrets. Consider and attempt to incorporate auditor feedback into subsequent versions of the ADS. Make a high-level summary of the feedback publicly available.
- Disclose to deployers results of the most recent performance evaluation, instructions for using and fine tuning the ADS, an explanation of circumstances in which a deployer’s fine tuning of the ADS requires them to assume the responsibilities of a developer, and any technical information needed to comply with the bill.
- Provide specified information received from audits of performance evaluation to deployers, including discrepancies from anticipated performance, unanticipated disparate impacts, and steps the deployer can take to mitigate discrepancies.
- Consider and attempt to incorporate feedback from auditors into development of subsequent versions of covered ADS.
- Ensure documentation provided to deployers is provided in a clear manner in a language commonly used with the developers, and that such documentation is retained for as long as the ADS remains deployed or available to potential deployers plus 10 years.
- Refrain from representing that an ADS can perform in a manner not substantiated by the results of the most recent performance evaluation, which under the bill constitutes false advertising.
- Designate at least one employee to oversee compliance with the bill’s requirements.
- As of January 1, 2027, requires deployers to:
- Before finalizing a consequential decision, provide subjects of the decision:
- Written notice with information about the covered ADS, including the personal characteristics it measures or assesses, sources of personal information collected from the subject, structure and format of the outputs, whether a natural person reviews the outputs or outcome before finalizing the consequential decision, and the subject’s right to opt out.
- A reasonable opportunity to opt out of the use of the ADS to make or facilitate the consequential decision.
- A deployer need not comply with (i) or (ii) if the subject is having a medical emergency or the deployer is subject to the federal Gramm-Leach-Bliley Act, which governs financial institutions.
- After finalizing a consequential decision, provide the subjects of the decision:
- Another written notice similar to the one described above, within 5 days.
- An opportunity to, within 30 days, correct incorrect personal information or appeal the outcome, as specified. If the exercise of either right would result in a different outcome, the deployer must rectify the outcome within 30 days of making this determination. If not, deployers must inform the subject. Deployers may deny a request to correct information but must provide the subject with an explanation for the basis of the denial and, for requests to correct the information, allow them a reasonable opportunity to request deletion of their personal information.
- Minimize their collection, use, retention, and sharing of personal information from subjects of consequential decisions.
- If the ADS impacts more than 5,999 people in a three-year period, contract with a third-party independent auditor to conduct an impact assessment on the covered ADS before January 1, 2030 and every three years thereafter.
- Assume the responsibilities of the developer if the deployer uses a covered ADS that impacts 5,999 or more people in a given three-year period and does not receive documentation relating to the developer-approved uses and performance evaluations for the covered ADS, or substantially modifies the ADS and uses it to impact 5,999 people in a three year period or otherwise makes the modified system available to potential deployers.
- Retain specified documentation for as long as the ADS is deployed plus 10 years.
- Designate at least one employee to oversee compliance with the bill.
- Comply with any additional requirements related to covered ADS promulgated by the California Privacy Protection Agency in duly adopted regulations.
- Before finalizing a consequential decision, provide subjects of the decision:
- Requires auditors conducting impact assessments every three years on behalf of deployers who use covered ADS that impact more than 5,999 people in that period to:
- Document: each developer-approved use of the covered ADS the deployer utilized; differences in observed and expected accuracy and reliability; whether disparate impacts resulted from the use of the ADS; whether the deployer used the ADS outside the scope of a developer-approved use; and whether the deployer assumed the responsibilities of a developer.
- Provide the results of the impact assessment to the developer and deployer.
- Make a high-level summary of the results of the impact assessment publicly available.
- Enables the Attorney General to obtain an unredacted performance evaluation or impact assessment, and to share that documentation with other enforcement entities as necessary for enforcement purposes. Such documentation is exempt from the California Public Records Act.
- Enables public prosecutors, the Civil Rights Department, and Labor Commissioner (with respect to employment-related violations) to bring a civil action against a developer or deployer to bring an action for injunctive or declaratory relief, reasonable attorney’s fees and costs, and a civil penalty of up to $25,000 per violation.
- Provides that a developer or deployer who contracts with a third party to comply with duties required under the bill, other than duties relating to auditing, is liable for the third party’s failure to comply with those duties.
- Does not:
- Apply to ADS that solely serve a cybersecurity function or operate aircraft in the national airspace.
- Apply to use of a consumer credit score.
- Limit rights, remedies, and penalties under any other law.
- Authorize use of an ADS, or disparate impacts and treatments, otherwise limited, restricted, or prohibited under any other law.
- Provides that in an action alleging a violation of the Unruh Civil Rights Act or FEHA in which an ADS is alleged to have committed or facilitated the violation, the defendant’s compliance with this bill is relevant to, but not conclusive of, the claim.
EXISTING LAW:
- Prohibits discrimination by “business establishments of every kind whatsoever” and guarantees all persons in California “full and equal accommodations, advantages, facilities, privileges, or services” regardless of race, sex, color, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, and other protected characteristics (Unruh Act). (Civil Code Section 51 (b).)
- Guarantees all persons in public schools equal rights and opportunities, regardless of their disability, gender, gender identity, gender expression, nationality, race or ethnicity, religion, sexual orientation, or any other characteristic that is contained in the definition of hate crimes. Specifies that the purpose of state law is to prohibit acts that are contrary to that policy and to provide remedies therefor. (Education Code Section 200 et seq.)
- Makes it an unlawful employment practice, except as provided, for an employer to discriminate against an employee or applicant for employment because of race, religious creed, color, national origin, ancestry, physical disability, mental disability, reproductive health decisionmaking, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, or veteran or military status. (Government Code Section 12940.)
- Governs the collection, disclosure, and protection of consumers’ personal financial information by financial institutions under federal law (Gramm-Leach-Bliley Act (GLBA)). (15 U.S.C. Section 6801 et seq.)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS: AB 1018 responds to the growing use of automated decision systems (ADS) in high-stakes sectors such as employment, housing, health care, and credit, where machine-driven tools are increasingly used to assist or replace human decisionmaking. While these systems offer efficiency and scale, numerous studies and real-world examples have shown that they can also replicate or exacerbate discriminatory outcomes—often in ways that are opaque to affected individuals and difficult to remedy after the fact. This bill represents the third and most comprehensive iteration of the author’s effort to create a regulatory framework around such systems. As explained by the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
The Growing Use of AI and Automated Decision Systems. Automated decision systems (ADS), powered by artificial intelligence (AI) and machine learning, are increasingly being used to make or influence consequential decisions across a wide range of sectors—including employment, education, housing, credit, healthcare, and public benefits. These systems can analyze vast datasets to produce outputs that inform or replace human judgment, such as predicting job applicant success, flagging fraud, determining creditworthiness, or evaluating eligibility for public services.
While these tools offer efficiency and scalability, their real-world deployment has outpaced legal and policy frameworks designed to safeguard fairness, transparency, and accountability. Numerous studies and investigative reports have shown that automated systems can entrench existing inequalities and produce biased or opaque results, especially when trained on historical data reflecting discriminatory practices or when operated without adequate oversight.
Systems marketed as “neutral” or “data-driven” have been found to replicate racially biased lending outcomes, under-assess health care needs in Black patients, misclassify college test-takers, and flag students with disabilities or limited internet access as “suspicious” during online proctoring. Real world examples include:
- Lending and Credit: Studies have shown that algorithmic lending tools may reproduce racially discriminatory patterns embedded in legacy credit data, resulting in higher denial rates for borrowers of color. (Kirchner, Lauren and Keegan, Emmanuel Martinez, The Secret Bias Hidden in Mortgage-Approval Algorithms, The Markup (Aug. 25, 2021), available at: https://themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms.)
- Online Proctoring: During the COVID-19 pandemic, thousands of students were subject to AI-powered surveillance tools that flagged “suspicious behavior” based on eye movement or internet lag. These systems raised privacy and equity concerns, particularly for students with disabilities and those with unstable internet access. (Washington Post, Students rebel over remote test monitoring during the pandemic (Nov. 12, 2020), available at: https://www.washingtonpost.com/technology/2020/11/12/test-monitoring-student-revolt.)
- Health Care: A major risk-scoring system used by hospitals was found to systematically underpredict the care needs of Black patients compared to white patients with the same health status, due to flawed use of health costs as a proxy for health outcomes. (Obermeyer, Ziad et al., Dissecting racial bias in an algorithm used to manage the health of populations, 366 Science 447 (Oct. 25, 2019),
available at: https://www.science.org/doi/10.1126/science.aax2342.) - Employment and Hiring: AI hiring platforms have been shown to reinforce gender stereotypes significant racial, gender, and intersectional biases in AI-driven resume screening tools. For example, resumes with white-associated names were preferred 85% of the time over those with Black-associated names. (Milne, Stefan, AI tools show biases in ranking job applicants’ names according to perceived race and gender, UW News (Oct. 31, 2024), available at: https://www.washington.edu/news/2024/10/31/ai-bias-resume-screening-race-gender/.)
Accountability Gaps and Inadequate Internal Oversight. Despite their growing influence, most ADS operate without meaningful external accountability. Current practices rely heavily on internal audits or self-certification by the very entities that develop and deploy these systems. As highlighted in academic literature, this internal model lacks the independence, access, and enforcement tools necessary to uncover or remediate harms, particularly those affecting marginalized communities. (Raji, Inioluwa Deborah et al., Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance, Proc. of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (AIES '22), Aug. 1–3, 2022, Oxford, United Kingdom, available at: https://doi.org/10.1145/3514094.3534181.) The academic literature further shows that first-party or consultant-run audits are frequently constrained by conflicts of interest, corporate control over disclosure, and narrow audit scopes. In high-profile examples, internal AI ethics teams at major tech firms were disbanded or censored after raising concerns about discriminatory or misleading products. Regulatory efforts like algorithmic impact assessments (AIAs) often default to internal processes without giving meaningful voice to affected communities or outside experts. (Id.) In short, AI policy has so far placed outsized focus on process and principles, but not enough on independent enforcement or systemic transparency. This gap creates a regulatory blind spot for high-risk ADS deployed in sectors that directly affect Californians’ rights, livelihoods, and well-being.
AB 1018 responds to this urgent need by establishing a comprehensive accountability framework for ADS used in consequential decisions. It targets high-impact uses, requires meaningful transparency to individuals affected by such decisions, and sets standards for performance evaluation, independent auditing, and public enforcement.
Third Time’s a Charm? AB 1018 marks the third iteration of the Legislature’s effort to regulate algorithmic decision-making in California, and it is by far the most comprehensive. This bill builds upon two prior measures—AB 331 (Bauer-Kahan, 2023) and AB 2930 (Bauer-Kahan, 2024)—each of which contributed foundational elements to the evolving statutory architecture aimed at preventing algorithmic discrimination and promoting fairness, transparency, and accountability in automated decision systems (ADS).
AB 331 (2023) introduced the conceptual framework, defining “automated decision tools,” identifying protected classes, and requiring developers and deployers to conduct annual “impact assessments” for systems used in consequential decisions. It relied heavily on internal documentation and risk analysis but offered limited enforceability or recourse for affected individuals.
AB 2930 (2024) refined these definitions and began shifting the focus toward the rights of individuals subject to algorithmic decisions. It required deployers to notify individuals when ADS were used to make impactful decisions and to offer opt-outs, explanations, and limited appeals. While it introduced important consumer-facing rights, AB 2930 stopped short of mandating independent audits, comprehensive disclosures, or integrating the regulatory scheme with existing civil rights law.
AB 1018 (2025) synthesizes and substantially expands the frameworks set forth in both earlier bills. It establishes a two-pronged compliance model: (1) developers must conduct rigorous performance evaluations and contract with independent auditors; and (2) deployers must ensure that subjects receive plain-language disclosures before and after consequential decisions are made or facilitated by an ADS. Individuals are guaranteed the right to opt out (with limited exceptions), correct erroneous personal information, and appeal decisions that may adversely affect them. Unlike its predecessors, AB 1018 contains detailed technical requirements, such as the treatment of “fine-tuning,” “substantial modifications,” and “developer-approved uses,” creating clear obligations and accountability mechanisms.
Perhaps most notably, AB 1018 explicitly integrates algorithmic governance into California’s civil rights infrastructure. It amends Civil Code Section 51 (Unruh Civil Rights Act) to state that compliance with the obligations set forth in Business and Professions Code Section 22756 is “relevant to, but not conclusive of” whether the use of an ADS violates the Unruh Act. This represents the first legislative attempt in California to directly tether algorithmic fairness obligations to one of the state’s core civil rights statutes, ensuring that algorithmic decision-making is not immune from anti-discrimination principles long established in California law.
The bill also introduces robust public enforcement authority, allowing the Attorney General, the Civil Rights Department, the Labor Commissioner (for employment-related uses), and local prosecutors to pursue violations. Daily noncompliance with audit submission requirements is treated as a separate offense, and civil penalties of up to $25,000 per violation create meaningful incentives for compliance.
In sum, AB 1018 reflects the maturation of the Legislature’s thinking in this space. While AB 331 laid the groundwork and AB 2930 introduced subject protections, AB 1018 delivers a cohesive and enforceable framework that blends transparency, technical rigor, and civil rights enforcement. If enacted, it would position California as a national leader in regulating algorithmic systems in a manner that balances innovation with fundamental fairness.
How this bill works. AB 1018 establishes a comprehensive regulatory framework for the use of ADS in consequential decisions affecting Californians. The bill targets systems that materially impact access to opportunities in high-stakes domains, such as employment, education, housing, credit, health care, legal services, and public benefits. Rather than banning or restricting particular technologies, the bill imposes procedural safeguards, transparency requirements, and accountability measures on developers and deployers of high-risk ADS. It also integrates this compliance framework into existing civil rights law and provides for enforcement by designated public agencies.
Covered Systems and Regulated Uses. This bill applies to “covered automated decision systems,” defined as machine-based systems that use artificial intelligence, machine learning, or statistical modeling to generate simplified outputs—such as classifications, scores, or recommendations—that are designed or used to assist or replace human discretionary decision-making. The system must be designed or used to make or facilitate a “consequential decision,” defined as a decision that materially impacts the cost, terms, quality, or accessibility of goods, services, or rights in any of sixteen specified domains, including but not limited to employment, education, housing, health care, family planning, credit, insurance, and elections.
The bill expressly excludes infrastructure and administrative tools such as spam filters, antivirus programs, calculators, datasets, and firewalls. Key definitions such as “developer-approved use,” “fine-tuning,” and “substantial modification” provide the basis for assigning duties and triggering compliance obligations throughout the system’s lifecycle.
Developer Obligations: Evaluation and Audit. Developers of covered ADS must conduct comprehensive performance evaluations to assess how the system functions in each approved use case. These evaluations are required:
- Prior to deployment for new systems;
- By January 1, 2027 for systems already in use;
- Annually thereafter;
- Following any substantial modification or any fine-tuning that materially alters the system’s output or use case.
Each evaluation must include:
- A statement of the ADS’s purpose and intended deployment contexts;
- A list of developer-approved uses;
- The system’s expected accuracy, reliability, and behavior across use cases;
- An analysis of any intended disparate treatment and the reasons for it;
- A projection of whether disparate impacts are likely, necessary, and avoidable;
- Documentation of any reported unanticipated disparate impacts;
- Mitigation strategies and safeguards.
Crucially, developers must contract with an independent third-party auditor to verify the adequacy of each evaluation. Auditors must be granted access to necessary information (subject to trade secret protections), and developers must publicly post a high-level summary of the auditor’s findings.
Deployer Duties: Notice, Transparency, and Individual Rights. Deployers of covered ADS are responsible for providing clear, timely, and actionable information to individuals whose rights may be affected by a consequential decision. Before finalizing such a decision, the deployer must provide the subject with plain-language written notice including:
- The fact that a covered ADS will be used;
- The system’s name, developer, and version;
- A description of how the ADS works, what data it uses, and whether a human will review the result;
- Whether the ADS use falls within a developer-approved context;
- An explanation of the subject’s rights, including opt-out, appeal, and correction rights.
Deployers must also give individuals a reasonable opportunity to opt out of ADS-based decision-making, unless the system is used for federal Gramm-Leach-Bliley Act-covered financial services or medical emergencies.
Following a consequential decision, the deployer must—within five days—provide a plain-language post-decision notice disclosing:
- What personal characteristics were used;
- What ADS output was generated and how it influenced the decision;
- Whether the output was reviewed by a human;
- Contact information for the deployer and any third-party managers;
- Instructions for submitting a correction or appeal request.
These disclosures must be provided in English and in any other language that the deployer regularly uses to communicate with subjects and accessible for individuals with disabilities.
Documentation and State Oversight. Developers and deployers must retain all relevant documentation for 10 years, including performance evaluations, notices, opt-out logs, audit records, appeal results, and any redacted materials related to trade secret claims.
Upon request, the Attorney General must be provided with unredacted copies of performance evaluations and impact assessments within 30 days. These records, when in the possession of the Attorney General, are:
- Exempt from disclosure under the Public Records Act;
- Protected by privilege and trade secret rules;
- Shareable with other public enforcement agencies only for regulatory purposes.
Failure to provide these materials or to complete required evaluations constitutes a daily violation.
Enforcement and Legal Remedies. This bill creates a public enforcement-only model, meaning that only specified government entities—not private individuals—may bring an action to enforce its provisions. Enforcement authority is granted to the Attorney General, district attorneys, city attorneys, county counsel, the Civil Rights Department, and the Labor Commissioner (the latter limited to employment-related uses of covered ADS). This multi-agency structure is intended to ensure broad geographic and subject matter coverage across the state, reflecting the diverse ways in which algorithmic decision systems may impact Californians’ rights.
Authorized entities may bring civil actions to seek a range of equitable and monetary remedies. These include injunctive and declaratory relief, reasonable attorneys’ fees, and civil penalties of up to $25,000 per violation. Importantly, the statute clarifies that each day on which a violation continues—such as operating a system without the required performance evaluation or audit—constitutes a separate violation. This daily accrual mechanism is designed to create meaningful incentives for prompt compliance and discourage prolonged noncompliance.
Additionally, the bill imposes vicarious liability on developers and deployers of ADS for the actions of contractors or agents who perform their duties under the Act, with a single exception: they are not liable for failures committed by independent third-party auditors. This approach prevents regulated entities from avoiding accountability by outsourcing core compliance responsibilities to third parties without oversight, while recognizing the independence and external function of mandated auditors.
Collectively, these enforcement provisions reflect a shift from the largely self-policing AI governance models seen in prior frameworks to one rooted in statutory enforceability, documented oversight, and public accountability. By authorizing a wide range of agencies to act and by attaching significant, repeatable penalties to violations, AB 1018 ensures that the duties it imposes on developers and deployers have real consequences if ignored.
Integration with the Unruh Civil Rights Act. This bill formally integrates the regulation of ADS into California’s broader civil rights enforcement structure by amending the Unruh Civil Rights Act (Civil Code Section 51) to address how algorithmic discrimination may be evaluated under existing law. Specifically, AB 1018 adds subdivision (i) to Section 51 of the Civil Code, providing that a developer’s or deployer’s compliance or noncompliance with the requirements of this chapter is “relevant to, but not conclusive of” whether a violation of the Unruh Act has occurred.
Existing Framework under the Unruh Civil Rights Act. The Unruh Civil Rights Act prohibits discrimination by “business establishments of every kind whatsoever” and guarantees all persons in California “full and equal accommodations, advantages, facilities, privileges, or services” regardless of race, sex, color, religion, ancestry, national origin, disability, medical condition, marital status, sexual orientation, and other protected characteristics (Civil Code Section 51 (b)). The Unruh Act applies broadly to most private businesses, including online platforms, retailers, and service providers.
To establish a violation under the Unruh Act, a plaintiff generally must show either:
- Intentional discrimination based on a protected characteristic; or
- A violation of the Americans with Disabilities Act.
Notably, California courts have historically been reluctant to find Unruh violations based on disparate impact alone without evidence of discriminatory intent (except for ADA claims)—especially where the challenged practice is neutral on its face.
AB 1018’s “Relevant but Not Conclusive” Standard. AB 1018 does not alter the substantive elements of an Unruh Act claim. Instead, it allows a court to consider whether the defendant complied with the procedural obligations imposed by this bill—such as providing pre- and post-decision notice, offering opt-out rights, conducting audits, and assessing disparate impacts—when deciding whether a covered entity engaged in unlawful discrimination under the Unruh Act.
The statutory language clarifies that such compliance is “relevant” but not “conclusive”—meaning that noncompliance with AB 1018 may support an inference of discriminatory practices, especially where algorithmic systems disproportionately impact protected groups and no mitigation steps were taken. Compliance, on the other hand, may weigh against a finding of unlawful discrimination by showing that the developer or deployer took proactive steps to detect and avoid disparate impacts.
This creates an evidentiary bridge between technical AI governance and California’s core civil rights protections. It does not, however, impose strict liability or create a new cause of action; courts retain full discretion in determining whether discrimination occurred based on the totality of the circumstances.
Departure from Prior Bills: AB 331 and AB 2930. This approach to integrating civil rights liability differs significantly from the two earlier legislative attempts to regulate algorithmic discrimination.
- AB 331 (2023) proposed a standalone statutory prohibition on algorithmic discrimination, enforced through a private right of action. It imposed a liability standard based on actual harm resulting from algorithmic discrimination but did not tie those harms to any existing civil rights framework. It also left open the question of whether facially neutral tools that caused disparate outcomes could be challenged if no intent was shown.
- AB 2930 (2024) introduced transparency and opt-out rights but did not contain any enforcement mechanism through existing civil rights statutes. While it required deployers to notify individuals when ADS were used in consequential decisions and to offer an alternative process upon request, it lacked any connection to broader anti-discrimination laws or the ability to use that noncompliance as evidence in an existing legal framework.
In contrast, AB 1018 does not create a new anti-discrimination standard, but rather embeds its requirements into the enforcement infrastructure of the Unruh Act. This reflects a more integrated and legally durable approach—anchoring procedural safeguards in a statute with longstanding interpretive precedent, while recognizing that technical compliance does not necessarily equate to nondiscriminatory outcomes.
By providing that compliance is a relevant but non-dispositive factor, AB 1018 ensures that algorithmic decisionmakers who ignore or undermine the law’s safeguards can be held accountable within existing civil rights doctrine, without fundamentally altering the substantive liability standard of the Unruh Act.
Opposition’s Concerns. A coalition of business organizations, led by the California Chamber of Commerce, has submitted a detailed opposition to AB 1018, raising concerns about the bill’s scope, feasibility, enforcement structure, and its potential effect on existing civil rights law. While many of these arguments reflect broader economic and policy views about how automated decision systems (ADS) should be governed in California, others raise specific implementation and legal questions within the scope of this Committee’s jurisdiction. Given the complexity and breadth of the systems and sectors the bill seeks to regulate, the author may wish to continue refining the measure as it moves forward in the legislative process, particularly to clarify definitions, mitigate compliance burdens, and coordinate with ongoing agency rulemaking efforts. This analysis will address a handful of the myriad additional concerns raised by the opposition.
Opposition’s Concern #1: Scope. Opponents argue that the scope of AB 1018 is overly broad, applying to businesses of all sizes and capturing a wide array of low-risk or general-purpose tools. As drafted, the bill defines a “covered ADS” as any system that assists or facilitates a consequential decision that materially impacts an individual’s access to essential opportunities or services, such as employment, education, or housing. The term “facilitates” has raised concern, as it could conceivably include tools used upstream in a decision-making process, including resume screening software, automated routing systems, or scheduling platforms. Although the bill limits coverage to systems that materially impact protected domains, they argue that the expansive definitions may create uncertainty for smaller entities or those using systems with indirect influence.
Opposition’s Concern #2: Impact on Anti-Discrimination Law. Opponents have raised concerns that AB 1018 alters the landscape of California discrimination law by (1) defining “disparate impact” and “disparate treatment” in statute; (2) appearing to create or expand Unruh Act liability based on noncompliance with procedural safeguards; and (3) deviating from standards established under the Fair Employment and Housing Act (FEHA) and judicial precedent. While many of these critiques are based on legitimate caution, several appear to reflect misreadings of the bill’s construction and the interaction between procedural compliance and liability standards.
Definitions & Liability. The opposition argues that the bill’s definitions of “disparate treatment” and “disparate impact” do not reflect the nuanced standards developed by courts over decades and may confuse or conflict with existing case law, jury instructions (e.g., CACI Nos. 2502 and 3060), and statutory frameworks like FEHA. But, AB 1018 defines “disparate treatment” and “disparate impact” solely for the purpose of performance evaluations and risk analysis—not for defining what constitutes unlawful discrimination. These definitions are included in proposed Business and Professions Code Section 22756, (i) and (j), not in the Civil Code, and are explicitly used to guide the risk assessment and audit processes required under the bill. Using “disparate impact” and “disparate treatment” ensures that developers and deployers are screening for the same types of harms that courts and regulators already recognize as forms of discrimination.
The definitions in AB 1018 are consistent with established federal and state jurisprudence:
- Disparate treatment is defined in the bill as “differential treatment of an individual or group of individuals on the basis of a protected characteristic”—a formulation in line with McDonnell Douglas Corp. v. Green (1973) 411 U.S. 792 and Guz v. Bechtel Nat. Inc. (2000) 24 Cal.4th 317, 354.
- Disparate impact is defined as “a differential effect on a group of individuals who share a protected characteristic.” This language reflects the conceptual threshold, not the full burden-shifting framework applied in litigation under FEHA or Title VII.
For example, Title VII of the Civil Rights Act, the Fair Employment and Housing Act (FEHA), and HUD’s fair housing regulations all rely on these concepts to identify discrimination that is either intentional (disparate treatment) or unjustified in its adverse effects (disparate impact). AB 1018 adopts these frameworks not to enforce liability, but to prompt early detection and correction of high-risk system behaviors. By naming these two core analytical concepts, the bill avoids ambiguity and provides developers, deployers, auditors, and enforcers with a shared vocabulary for identifying and analyzing discriminatory outcomes. The performance evaluation must address whether disparate treatment is intended and necessary, and whether disparate impacts are reasonably foreseeable, avoidable, and mitigated. These concepts become the foundation for evidence-based, proactive oversight, rather than ad hoc or inconsistent analysis.
The opposition correctly notes that intentional discrimination is generally required under Unruh. This was affirmed by the California Supreme Court in Harris v. Capital Growth Investors XIV (1991) 52 Cal.3d 1142, where the Court held that disparate impact alone does not establish a violation of Section 51 absent a showing of intentional discrimination. The opposition’s concerns appear overblown: AB 1018 does not disturb this precedent. Instead, the proposed Civil Code Section 51(i) merely allows a court to consider compliance with procedural safeguards (such as conducting a performance evaluation or audit) when evaluating whether intent may be inferred. This is functionally no different than courts considering whether a business had notice of potential discriminatory outcomes or failed to act in response to known disparities. This structure does not permit liability to be based solely on a failure to conduct an assessment. The “relevant, but not conclusive” standard preserves judicial discretion and leaves intact existing pleading and evidentiary burdens.
Nevertheless, to address concerns that AB 1018 could be interpreted as altering substantive standards for proving discrimination under existing civil rights laws, the author proposes amendments to clarify the scope and effect of the bill’s provisions. Specifically, the amendments make clear that the definitions of “disparate treatment” and “disparate impact” in the bill are intended solely to guide internal compliance, evaluation, and documentation requirements under the Business and Professions Code, and are not intended to modify the legal standards or burdens of proof applied by courts in actions under the Unruh Civil Rights Act, the Fair Employment and Housing Act, or other anti-discrimination laws. Additionally, the amendments clarify that in a civil action brought under the Unruh Act, a developer’s or deployer’s failure to comply with AB 1018’s procedural obligations—such as conducting a performance evaluation or audit—may be considered by a court, but does not by itself establish intent or liability. The bill thus preserves existing legal standards while allowing courts to consider procedural diligence as one of many factors in evaluating a claim. These proposed amendments are included supra.
Opposition’s Concern #3: Notice, Opt-out, and Appeal Requirement. These procedural rights are intended to ensure that individuals understand when and how ADS are used to make high-stakes decisions that affect them. However, opponents argue that the bill’s notice and opt-out framework is not practicable in dynamic, high-volume settings, such as staffing agencies, healthcare triage, or fraud detection. While the bill includes a limited emergency exemption and requires opt-out only when “technically feasible,” they argue that it may still be interpreted to require burdensome individualized disclosures and alternative manual processes.
Opposition’s Concern #4: Enforcement Framework. Opponents “appreciate” the absence of a private right of action, but nevertheless argue that AB 1018’s enforcement provisions are overly punitive, vague, and duplicative. Specifically, they raise three key concerns: (1) the bill authorizes enforcement by multiple public entities rather than a single agency; (2) it imposes civil penalties of up to $25,000 per violation without requiring a showing of actual harm; and (3) the term “violation” is undefined, raising fears of penalty stacking. They suggest limiting enforcement to the Attorney General and incorporating a cure period for first-time violations. In response, it is important to recognize that California law routinely authorizes multi-agency public enforcement of consumer protection and civil rights statutes—particularly where violations may occur at the local level or affect broad populations. For example, the Unfair Competition Law (Business and Professions Code Section 17200 et seq.), the Consumer Legal Remedies Act (Civil Code Section 1750 et seq.), and the False Advertising Law (Business and Professions Code Section 17500 et seq.) all authorize enforcement not only by the Attorney General, but also by district attorneys, county counsel, and city attorneys (not to mention private enforcement in certain instances). These statutes reflect the state’s long-standing commitment to decentralized public enforcement in areas affecting consumer welfare and civil rights. In this context, AB 1018’s multi-agency enforcement approach is not unusual. Because the bill regulates a wide range of sectors—housing, employment, education, health care, legal services—it makes sense to empower agencies that are best positioned to identify harm within their jurisdiction. For example, the Labor Commissioner has expertise in enforcing employment-related protections and may be more effective in detecting unlawful ADS use in hiring platforms than a centralized entity.
The $25,000 per violation penalty amount mirrors the structure in AB 2930 (2024), which set the same maximum penalty for algorithmic discrimination. Here, that penalty is triggered by failures to comply with technical requirements of this measure (separate liability could still attach under Unruh for actual discrimination). This per penalty amount is relatively high, especially considering the vagueness in the term “violation.” The bill currently does not specify whether a “violation” means each missing audit, each technical flaw in an evaluation, or each affected individual. Nor does it distinguish between willful misconduct and good-faith error. Without further guidance, opposition argues, the penalty framework could be interpreted too broadly or inconsistently. The fact that AB 1018 does not require a showing of actual harm to impose penalties is consistent with its preventive framework. The bill’s primary goal is not to punish developers or deployers post hoc, but to establish baseline diligence requirements that reduce the risk of discriminatory outcomes before they occur. In that spirit, it is appropriate for penalties to attach to failures in documentation, auditing, and notice—not just discriminatory outcomes—so long as those obligations are clearly defined and not subject to arbitrary enforcement.
Author’s Amendments. The bill is proposed to be amended to add new subdivision (s) to Section 22756 of the Business and Professions Code as follows:
(s) The definitions of ‘disparate impact’ and ‘disparate treatment’ provided in subdivisions (i) and (j) of this section are intended solely for purposes of internal compliance, risk assessment, and documentation required by this chapter. These definitions shall not be construed to modify or supersede any standard, burden of proof, or element of a claim under the Unruh Civil Rights Act (Section 51 of the Civil Code), the Fair Employment and Housing Act (Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code), Title VII of the Civil Rights Act of 1964 (42 U.S.C. Sec. 2000e et seq.), or any other applicable civil rights law.
In addition, proposed Civil Code Section 51(i) will be amended to clarify that procedural noncompliance alone does not establish intent, as follows:
In an action alleging a violation of this section in which the defendant’s development, modification, or use of an automated decision system, as defined in Section 22756 of the Business and Professions Code, is alleged to have caused or facilitated the violation, the extent to which the defendant complied with Chapter 24.6 (commencing with Section 22756) of Division 8 of the Business and Professions Code is relevant to, but not conclusive of, whether the defendant violated this section. A failure to comply with Chapter 24.6 shall not, by itself, give rise to a presumption of unlawful intent.
ARGUMENTS IN SUPPORT: A broad coalition of civil society and labor organizations jointly write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile, to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones. Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives. Specifically, the legislation:
Requires these tools to be tested before they are used on the public: Requires that people who make and use these tools test them to make sure they do not create harm and comply with our existing rights to non-discrimination before they are sold and used on the public. It ensures that these tests are verified by an independent third party.
Provides a notice to people that this tool will be used to make a critical decision about their life: Provides people the information they need to understand where these tools are showing up in their lives and how they’ll be used to determine their housing, healthcare, and job outcomes.
Provides an explanation to people who were subject to a decision made with these tools: Provides people an explanation of what the tool did, what personal information it used about them to make the decision and what role the tool played in making the decision.
Ensures that every day people have more control over how these tools are used in their daily lives: Through this bill, people will have the right to opt out of the use of an ADS tool in a critical decision about them; they will be able to correct information that the tool used to make the decision if it is inaccurate; and they will have the right to appeal the decision.
Pre-deployment testing protects Californians by ensuring that businesses understand the risks and accuracy of the systems they’re purchasing and deploying to make consequential decisions about Californians. The strong notice and explanation provisions in this bill are a critical first step for consumers to know how these tools are being used and to be able to exercise their right to correct their data or appeal a decision. These protections are more important than ever as we see a Federal pullback from safety guardrails on AI and as other states and jurisdictions pass laws recognizing the danger inherent in letting these systems enter into our lives without clear protections and safeguards for the public.
ARGUMENTS IN OPPOSITION: A coalition of industry trade organizations, led by California Chamber of Commerce, writes that the bill:
. . .fails to focus on the stated objective of requirements on high-risk automated decision systems (ADS) to the detriment of every industry in the state of California, and from small business to large. Instead, the bill broadly targets businesses of all sizes, across every industry, and regulates even lower risk applications of ADS, including those that are already in use, and unnecessarily alters what constitutes discrimination under California law. Such overreach not only exposes smaller businesses to significant – if not devastating – liability even for mere errors that caused no harm to consumers, but it would also hinder many beneficial uses of ADS, including but not limited to: enabling faster approvals and expanded access to credit; enhancing real-time fraud detection; fostering job creation and new industries; improving efficiency to help level the playing field between small and large businesses; addressing major societal challenges such as bias and discrimination, economic inequality, climate change, injustices in the criminal system, disaster relief, and humanitarian aid; and advancing new treatments for previously incurable diseases.
To be clear, whether decisions are being made by humans from start to end, or a byproduct of using or incorporating new technologies in the decision-making process, we take our responsibility to not discriminate with the utmost seriousness. We believe that algorithmic discrimination, or discrimination that results from AI-enabled technologies, is already prohibited under our anti-discrimination laws because our laws are rights-based and not technology-specific. We also agree that companies need to take care to reduce bias and discrimination in decisions that have legal impact on the provision or denial of fundamental rights or essential opportunities and services, which is why so many of them already conduct impact assessments.
But it would be incredibly short-sighted for regulation to stifle innovation when alternative (human-decision driven) systems may be equally, if not more, flawed, and when properly developed and deployed ADS can enhance fairness and accountability. While ADS may pose unique challenges in terms of bias, they also pose unique advantages to combatting it as well: for example, ADS decision-making processes can be more transparent and traceable than the exercise of human discretion. The structured nature of these tools offer opportunities for detection and correction that can be more challenging to come by in human decision-making processes where human bias is often more subtle and harder to detect. Unfortunately, we believe AB 1018 will have an undesired chilling effect on the technology and make it that much harder to develop the very tools that can help combat bias in decisionmaking.
As introduced, AB 1018 goes far beyond ensuring developers and deployers of these technologies act responsibly to adhere to existing anti-discrimination protections and veers into an indirect restriction upon the usage of the technology itself, discouraging technological innovation and usage by making it so onerous and risky that businesses are realistically pushed back toward the alternative, human driven process – which we know based on historical evidence will not help eliminate bias and discrimination in any way.
REGISTERED SUPPORT / OPPOSITION:
Support
Techequity Action (co-sponsor)
SEIU California (co-sponsor)
Acce Action (alliance of Californians for Community Empowerment)
American Federation of Musicians Local 47
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Employment Lawyers Association
California Federation of Labor Unions, AFL-CIO
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, Aft, Afl-cio
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center (EPIC)
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Justice2jobs Coalition
Kapor Center
LA Defensa
Laane (Los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Northern California District Council of the International Longshore and Warehouse Union (ILWU)
Oakland Privacy
Orange County Employees Association
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Transparency Coalition.ai
UDW/AFSCME Local 3930
UFCW - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
Support if Amended
California Nurses Association
Opposition
Abundance Institute
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
Association of California Life & Health Insurance Companies
Association of National Advertisers
California Chamber of Commerce
Chamber of Progress
Civil Justice Association of California (CJAC)
College Board
Computer and Communications Industry Association
Consumer Data Industry Association
Independent Insurance Agents & Brokers of California, INC.
Internet.works
National Association of Mutual Insurance Companies
Pacific Association of Domestic Insurance Companies
Personal Insurance Federation of California
Public Risk Innovation, Solutions, and Management (PRISM)
Security Industry Association
Silicon Valley Leadership Group
Software Information Industry Association
TechCA
TechNet
Opposition Unless Amended
Advanced Medical Technology Association (ADVAMED)
America's Physician Groups
American College of Obstetricians & Gynecologists - District Ix
Calbroadband
Calcom Association
California Association of Health Plans
California Bankers Association
California Community Banking Network
California Credit Union League
California Hospital Association
California Life Sciences
California Medical Association (CMA)
California Mortgage Bankers Association
California Radiological Society
CTIA
Kaiser Permanente
Mortgage Bankers Association
Securities Industry and Financial Markets Association
Sutter Health
Other
Business Software Alliance
County Welfare Directors Association of California
Analysis Prepared by: Shiran Zohar / JUD. / (916) 319-2334
Date of Hearing: April 22, 2025
Fiscal: Yes
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
ABPCA Bill Id:AB 1018 (
Author:Bauer-Kahan) – As Amended Ver:April 10, 2025
SUBJECT: Automated decision systems
SYNOPSIS
Automated Decision Systems (ADS) typically use artificial intelligence (AI) that produce simplified outputs – such as scores, classifications, or recommendations – to assist or replace human discretionary decisionmaking. Often these decisions have low-stakes; sometimes, however, they can meaningfully impact the public’s rights, opportunities, or access to critical resources or services. ADS can process enormous datasets, identify hidden patterns, and make decisions with efficiency and scale that vastly exceeds human capabilities. But relying on ADS to make consequential decisions can be hazardous if the systems are not trained carefully or tested thoroughly: the datasets they are trained on are often unrepresentative or contaminated with bias, the inferences they draw from those datasets are often inscrutable, and these systems can fail to accurately account for the complexity of human behavior. Without human-centered oversight, particularly in consequential contexts such as employment, housing, healthcare, and criminal justice, these impacts can be irreparable.
In 2022, the White House Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights, which identifies five principles that should “guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence.”[1] These principles would (1) protect the public from unsafe and ineffective ADS, (2) prevent algorithmic discrimination, (3) protect data privacy, (4) give subjects notice and explanation when ADS are used, and (5) provide for human alternatives, consideration, and fallback. The Blueprint calls for rigorous testing, monitoring, and independent evaluation of ADS to ensure that ADS are at least as fair and trustworthy as the humans they are displacing.
This bill is the author’s third attempt to realize that vision. In broad strokes, the bill is similar to its predecessors: it seeks to create a comprehensive transparency regime for developers and deployers of ADS that are used for consequential decisionmaking. The author’s goal is to prevent algorithmic discrimination on the basis of protected characteristics and to ensure that ADS can be trusted to perform reliably and accurately. The bill would require deployers of ADS to give subjects of consequential decisions notice, an ability to opt out of the use of the ADS, a chance to correct any personal information used by the ADS to make the decision, and a right to appeal the outcome of the decision. The bill also provides for independent evaluation by third-party auditors – a substantial but controversial provision not found in the prior bills. Enforcement of the bill may be undertaken by the Attorney General, public prosecutors, the Civil Rights Department, or Labor Commissioner, within their respective jurisdictions.
The bill is co-sponsored by SEIU California and TechEquity and is supported by a broad coalition of civil society, consumer protection, and labor organizations. Supporters argue that the bill establishes common-sense guardrails to prevent inaccurate or discriminatory decisions by ADS – protections that are especially needed after the federal government’s retreat from AI regulation.
The bill is opposed by a coalition of industry trade organizations, led by California Chamber of Commerce, who set forth several concerns relating to the bill’s breadth, workability, and costs. A number of other businesses and organizations take an oppose-unless-amended position.
If passed by this Committee, this bill will next be heard by the Judiciary Committee.
THIS BILL:
- Defines key terms, including:
- “Automated decision system,” or ADS, means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons. The term does not include a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
- “Consequential decision” means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
- Employment-related decisions, as defined.
- Education and vocational training as they relate to specified categories.
- Housing and lodging as they relate to rentals or short-term housing and lodging, home appraisals, rental subsidies, and publicly supported housing.
- Essential utilities: electricity, heat, water, transportation, and municipal trash and sewage services.
- Family planning, adoption services, reproductive services, and assessments related to child protective services.
- Health care and health insurance.
- Financial services.
- The criminal justices system with respect to pretrial release, sentencing, and alternatives to incarceration.
- Legal services, private arbitrations, and mediation.
- Elections, as they relate to specified categories.
- Access to government benefits or services or assignment of penalties by a government entity.
- Places of public accommodation, as specified.
- Insurance.
- Internet and telecommunications access.
- “Covered ADS” means an ADS that is designed or used to make or facilitate a consequential decision.
- “Deployer” means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
- “Developer” means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
- “Developer-approved use” means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision, and includes any reasonably foreseeable fine tuning of the covered ADS.
- “Disparate impact” means a differential effect on a group of individuals who share a protected characteristic.
- “Disparate treatment” means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
- “Express consent” means affirmative written authorization granted in response to a notice that meets specified requirements.
- “Fine-tune” means to adjust the model parameters of an ADS through exposure to additional data.
- “Personal information” has the same meaning as defined in the California Consumer Privacy Act.
- “Protected characteristic” means a characteristic listed in the Unruh Civil Rights Act.
- “Substantial modification” means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs, but does not include modifications resulting from fine-tuning.
- “Trade secret” has the same meaning as provided under the Uniform Trade Secrets Act.
- Requires developers to:
- Conduct initial performance evaluations on ADS either before they are first deployed or, if they were deployed before January 1, 2026, by January 1, 2027.
- Conduct subsequent performance evaluations on ADS no more than a year apart, including following any substantial modification or fine-tuning.
- In conducting a performance evaluation, for each developer-approved use, document:
- The expected accuracy and reliability of the covered ADS.
- Whether any disparate treatment is intended to occur, whether such treatment is necessary, and whether alternatives were considered.
- Whether the covered ADS is reasonably likely to result in disparate impacts, whether these impacts are necessary, and whether alternatives were considered.
- Whether measures were taken to mitigate the risk of unanticipated disparate impacts, including any unanticipated disparate impacts reported by deployers.
- Any reasonably foreseeable effects of fine tuning, as specified.
- Contract with an independent third-party auditor to assess the developer’s compliance with requirements for performance evaluations. Provide the auditor with necessary documentation, which may be redacted to protect trade secrets. Consider and attempt to incorporate auditor feedback into subsequent versions of the ADS. Make a high-level summary of the feedback publicly available.
- Disclose to deployers results of the most recent performance evaluation, instructions for using and fine tuning the ADS, an explanation of circumstances in which a deployer’s fine tuning of the ADS requires them to assume the responsibilities of a developer, and any technical information needed to comply with the bill.
- Provide specified information received from audits of performance evaluation to deployers, including discrepancies from anticipated performance, unanticipated disparate impacts, and steps the deployer can take to mitigate discrepancies.
- Consider and attempt to incorporate feedback from auditors into development of subsequent versions of covered ADS.
- Ensure documentation provided to deployers is provided in a clear manner in a language commonly used with the developers, and that such documentation is retained for as long as the ADS remains deployed or available to potential deployers plus 10 years.
- Refrain from representing that an ADS can perform in a manner not substantiated by the results of the most recent performance evaluation, which under the bill constitutes false advertising.
- Designate at least one employee to oversee compliance with the bill’s requirements.
- As of January 1, 2027, requires deployers to:
- Before finalizing a consequential decision, provide subjects of the decision:
- Written notice with information about the covered ADS, including the personal characteristics it measures or assesses, sources of personal information collected from the subject, structure and format of the outputs, whether a natural person reviews the outputs or outcome before finalizing the consequential decision, and the subject’s right to opt out.
- A reasonable opportunity to opt out of the use of the ADS to make or facilitate the consequential decision.
- A deployer need not comply with (i) or (ii) if the subject is having a medical emergency or the deployer is subject to the federal Gramm-Leach-Bliley Act, which governs financial institutions.
- After finalizing a consequential decision, provide the subjects of the decision:
- Another written notice similar to the one described above, within 5 days.
- An opportunity to, within 30 days, correct incorrect personal information or appeal the outcome, as specified. If the exercise of either right would result in a different outcome, the deployer must rectify the outcome within 30 days of making this determination. If not, deployers must inform the subject. Deployers may deny a request to correct information but must provide the subject with an explanation for the basis of the denial and, for requests to correct the information, allow them a reasonable opportunity to request deletion of their personal information.
- Minimize their collection, use, retention, and sharing of personal information from subjects of consequential decisions.
- If the ADS impacts more than 5,999 people in a three-year period, contract with a third-party independent auditor to conduct an impact assessment on the covered ADS before January 1, 2030 and every three years thereafter.
- Assume the responsibilities of the developer if the deployer uses a covered ADS that impacts 5,999 or more people in a given three-year period and does not receive documentation relating to the developer-approved uses and performance evaluations for the covered ADS, or substantially modifies the ADS and uses it to impact 5,999 people in a three year period or otherwise makes the modified system available to potential deployers.
- Retain specified documentation for as long as the ADS is deployed plus 10 years.
- Designate at least one employee to oversee compliance with the bill.
- Comply with any additional requirements related to covered ADS promulgated by the California Privacy Protection Agency in duly adopted regulations.
- Before finalizing a consequential decision, provide subjects of the decision:
- Requires auditors conducting impact assessments every three years on behalf of deployers who use covered ADS that impact more than 5,999 people in that period to:
- Document: each developer-approved use of the covered ADS the deployer utilized; differences in observed and expected accuracy and reliability; whether disparate impacts resulted from the use of the ADS; whether the deployer used the ADS outside the scope of a developer-approved use; and whether the deployer assumed the responsibilities of a developer.
- Provide the results of the impact assessment to the developer and deployer.
- Make a high-level summary of the results of the impact assessment publicly available.
- Enables the Attorney General to obtain an unredacted performance evaluation or impact assessment, and to share that documentation with other enforcement entities as necessary for enforcement purposes. Such documentation is exempt from the California Public Records Act.
- Enables public prosecutors, the Civil Rights Department, and Labor Commissioner (with respect to employment-related violations) to bring a civil action against a developer or deployer to bring an action for injunctive or declaratory relief, reasonable attorney’s fees and costs, and a civil penalty of up to $25,000 per violation.
- Provides that a developer or deployer who contracts with a third party to comply with duties required under the bill, other than duties relating to auditing, is liable for the third party’s failure to comply with those duties.
- Does not:
- Apply to ADS that solely serve a cybersecurity function or operate aircraft in the national airspace.
- Apply to use of a consumer credit score.
- Limit rights, remedies, and penalties under any other law.
- Authorize use of an ADS, or disparate impacts and treatments, otherwise limited, restricted, or prohibited under any other law.
- Provides that in an action alleging a violation of the Unruh Civil Rights Act or FEHA in which an ADS is alleged to have committed or facilitated the violation, the defendant’s compliance with this bill is relevant to, but not conclusive of, the claim.
EXISTING LAW:
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act. (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act, which provides that all persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever. (Civil Code § 51(b).)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
- Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
- Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civ. Code § 3426.1(d).)
- Establishes the California Privacy Protection Agency (Privacy Agency) and vests it with full administrative power, authority, and jurisdiction to implement and enforce the California Consumer Privacy Act (CPPA) of 2018. (Civ. Code § 1798.199.10.)
- Requires the California Privacy Protection Agency to issue regulations governing access and opt-out rights with respect to businesses’ use of automated decisionmaking technology. (Civ. Code § 1798.185.)
- Establishes prohibitions on false and deceptive advertising. (Bus. & Prof. Code § 17500 et seq.)
COMMENTS:
1) Author’s statement. According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
2) Automated Decision Systems. As a general matter, ADS use AI to make life-impacting decisions. AI refers to the mimicking of human intelligence by artificial systems such as computers. AB 2885 (Bauer-Kahan, Stats. 2024, Ch. 843) defined the term as “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.” AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; unlike other computer functions, however, AI is able to accomplish tasks that are normally performed by humans.
Most modern AI tools are created through a process known as “machine learning.” Whereas traditional programming is manually written by a programmer and can only follow pre-defined rules, with machine learning algorithms, programmers do not determine the rules the algorithm will follow. Instead, the AI is allowed to automatically explore the structure of data and infer the relationship between the data and desired outputs.[2] Machine-learning algorithms are often criticized for being “black boxes” that generate predictions and outcomes that cannot be clearly explained.[3] “It’s often observed in the field that the most powerful models are on the whole the least intelligible, and the most intelligible are among the least accurate.”[4]
The process of exposing a naïve AI to data is known as “training.” The algorithm that an AI develops during training is known as its “model.” Training is the secret sauce of machine learning. At its core, training is an optimization problem wherein a model attempts to identify specific parameters – “weights” – that minimize the difference between predicted outcomes and actual outcomes. How an input is transformed into an output depends on the specific algorithm that is developed by a model. Once trained, AI can be used to process new, never-before-seen data.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from generative AI (GenAI,) which are trained on massive datasets in order to produce detailed text, images, audio, and video. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that is trained on the written contents of the internet.[5] When Netflix suggests content to a viewer, its recommendation is produced by predictive AI that is trained on the viewing habits of Netflix users.[6]
ADS typically use predictive AI to generate outputs that influence real-world outcomes.[7] Existing law defines an ADS to include “a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.”[8]
When the stakes of these outcomes are high, such as predicting a person’s risk of defaulting on a loan or committing a crime, the use of an ADS can cut both ways: these systems can create efficiencies, ensure uniformity, and detect intricate patterns among massive datasets, leading to incredibly beneficial applications and breakthroughs.[9] On the other hand, use of ADS can also raise significant questions of fairness, accountability, and transparency. Government Code section 11546.45.5(a)(4) defines “high-risk” ADS as those that are “used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.”
This bill incorporates the existing definition of ADS and applies to any such system used to make or facilitate a “consequential decision,” defined under the bill as a decision that materially impacts the cost, terms, quality, or accessibility of several enumerated categories with respect to a natural person.
3) Algorithmic discrimination. There is a well-known saying in computer science: “garbage in, garbage out.” The performance of an ADS is directly impacted by the quality, quantity, and relevance of the data used to train it.[10] If the data used to train the ADS is biased, the tool’s outputs will be similarly biased. Over the past thirty years, several industries have been forced to contend with this fact as they have attempted to introduce ADS into their workflows. Specific examples follow.
Child welfare: In 2016, Allegheny County, Pennsylvania adopted a family screening tool to predict which families should be investigated by social workers for possible removal of maltreated children. The tool was trained only on data from poorer families who used public services such as Medicaid. Because its training dataset lacked examples of wealthier families, the tool disproportionately targeted poorer families.[11]
Credit and loan approval. Financial tools that utilize ADS are similarly susceptible to bias and discrimination. An investigation by The Markup (and co-published by the Associated Press) revealed that in 2019, lenders were more likely to deny home loans to people of color; in particular, lenders were 40 percent more likely to turn down Latino applicants for loans, 50 percent more likely to deny Asian/Pacific Islander applicants, and 70 percent more likely to deny Native American applicants than similar white applicants. Lenders were 80 percent more likely to reject Black applicants than similar white applicants. In every case, the prospective borrowers of color looked almost exactly the same on paper as the white applicants, except for their race. According to the report, mortgage-approval algorithms play a major role in perpetuating these inequities.[12]
Education. Professor Broussard’s article in the New York Times details how “the International Baccalaureate—a global program that awards prestigious diplomas to high school students—canceled its usual in-person exams because of the [COVID-19] pandemic” and instead “used an algorithm to ‘predict’ student grades based on an array of student information, including teacher-estimated grades and past performance by students in each school.”[13] Tens of thousands of students, surprised to find out they failed, protested the results. “High-achieving, low-income students were hit particularly hard: many took the exams expecting to earn college credit with their scores and save thousands of dollars on tuition.”[14]
Employment. It is no secret that people of various races, genders, and cultures are not distributed equally throughout the workforce. An ADS that is trained on historical data to make hiring decisions will be predisposed to maintain the ratios it is trained on; as described by Aditya Malik, the Founder and CEO of Valuematrix.ai:
Generative AI, for all its grandeur, has the potential to perpetuate latent biases inherited from human creators. A disconcerting echo of historical prejudices may inadvertently seep into the algorithms. Imagine a scenario where previous senior managers, driven by biases of gender, age, faith or race, rejected candidates for misguided reasons. The AI, if not vigilantly curated, might misconstrue these patterns as indicators of incompetence, thus exacerbating the exclusion of qualified candidates from underrepresented backgrounds.[15]
This was notoriously experienced by Amazon, who considered automating their hiring practices in the early 2010s. They opted against this approach in 2015 when they realized that their ADS-enabled system was not rating candidates in a gender-neutral way. In fact, their system was excluding women from the pool of acceptable candidates because it had been trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of inequities across the tech industry.[16]
ADS are being employed in the workforce as well. For example, “[a] company installed AI-powered cameras in its delivery vans in order to evaluate the road safety habits of its drivers, but the system incorrectly penalized drivers when other cars cut them off or when other events beyond their control took place on the road. As a result, drivers were incorrectly ineligible to receive a bonus.”[17]
Healthcare. When ADS are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. In 2007, an ADS was developed to help doctors estimate whether it was safe for people who had delivered previous children through cesarean section to deliver subsequent children vaginally – a risky procedure. The ADS considered various health relevant factors as it made its decision, such as the woman’s age, her reason for the previous cesarean, and how long ago the cesarean had been performed. However, a 2017 study found that the ADS was biased; it predicted Black and Latino people were less likely to have a successful vaginal birth after a cesarean than non-Hispanic white women. As a result, doctors performed more cesareans on Black and Latino people than on white people.[18] These discrepancies perpetuate historical biases – Black Americans, for example, have historically received a lower standard of healthcare than their white counterparts.[19]
Similarly, a 2019 study found strong racial bias in a system used to identify patients with a high risk of adverse health outcomes. The ADS assigned Black patients lower likelihoods than equally at-risk White patients. The authors found that this happened because the ADS was designed to predict healthcare costs instead of needs. Because the healthcare system has historically spent less on care for Black patients than White patients for the same health conditions, the ADS was, in essence, issuing a prediction that mirrored and perpetuated past discrimination.[20]
Housing. A recent ProPublica article found that tenant screening companies compile information on renters beyond credit reports – including criminal background, evictions filings, medical debt, and student loans – and then use algorithms that “try to predict how risky it is to rent to a potential tenant based on characteristics they share with other tenants” and “assign applicants scores or provide landlords a yes-or-no recommendation.”[21] In 2023, concerns about these tools led several Attorneys General – California’s Rob Bonta included – to write in a comment letter to the Federal Trade Commission and the Consumer Financial Protection Bureau:
Landlords and tenant screening companies increasingly rely on problematic screening algorithms that combine various information to generate a single score or result indicating how “safe” it would be to rent to a prospective tenant. These algorithms lack transparency and can be inaccurate, and can have a discriminatory impact on underserved communities. The attorneys general recommend, among other things, requiring that tenant screening companies disclose their reliance on algorithms and screen algorithm models for bias against protected classes and prohibiting the use of certain types of records in screening reports.[22]
Sentencing and bail decisions. ADS are frequently used to inform sentencing and bail decisions. These tools are trained using historical data, and the predictions they make can therefore reflect historical bias. A 2016 ProPublica study dove into the use of one such tool – COMPAS – in Broward County, Florida.[23] The study determined that Black defendants were far more likely than white defendants to be incorrectly judged to be at a higher risk of recidivism, while white defendants were more likely than Black defendants to be incorrectly flagged as low risk.
The for-profit company that developed this tool, Northpointe, does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. These discrepancies mirror historical injustices perpetuated against Black Americans by California’s criminal justice system.[24] The use of a proprietary algorithm by government actors also raises significant due process questions, as it becomes difficult for individuals to understand, let alone challenge, arbitrary government deprivations of life, liberty, or property.[25]
Structural effects of discriminatory ADS: ADS becoming more pervasive across institutions raises the risk of a feedback loop that compounds and reinforces discrimination. As Borocas et al. write in Fairness and Machine Learning, “predictive systems have the effect of transferring advantages from one phase of life to the next, and one generation to the next.” This can compound and amplify injustices: “individuals are subject to a series of decisions over the course of their lives, and the effects of these decisions both accumulate and compound over time. When a person receives (or is denied) one opportunity, they are likely to appear more (or less) qualified at their next encounter with a predictive system.”[26]
4) Unsafe or ineffective systems. In addition to discriminatory outcomes, some ADS are unsafe or simply ineffective regardless of who the subjects of the prediction are. A few common types of flawed ADS are described below.
Spurious correlations. Accurate predictions may nevertheless lead to bad decisions. In one example, a hospital trained AI models on a dataset of 15,000 pneumonia patients in order to develop a model that could identify which pneumonia patients were at the greatest risk in order to triage new patients. During testing, it was discovered that one of the most accurate models recommended outpatient status for asthmatics – a life-threateningly dangerous error based on an accurate statistical correlation: asthmatics are less likely to die from pneumonia than the general population precisely because asthma is such a serious risk factor that asthmatics automatically get elevated care.[27]
In other cases, the correlations that machine-learning ADS may rely on have little to do with the attributes they purport to measure. Some hiring tools record videos of applicants responding to pre-recorded questions and attempt to assess their fitness for a job based on their facial expression and intonation. Such tools can be gamed by making simple changes to the subject’s appearance, such as wearing glasses, or the background of the room, such as adding more books on the bookshelf – leading to increased scores. Journalist Hilke Schellmann found she was able to obtain consistently high scores on one hiring ADS despite responding by reading an irrelevant Wikipedia entry in German.[28]
Irrelevant datasets. “AI reflects its training data. It learns patterns about the people who make up the data, and the decisions made by AI reflect these patterns. But when the decision subjects come from a population with different characteristics than those in the training data, the model’s decisions are likely to be wrong.”[29] For instance, the Ohio Risk Assessment System was trained on data from just 452 defendants from Ohio, but has been deployed in several other states, despite its small and unrepresentative dataset.[30]
Snake oil. Some tools, although marketed as automating precision, are simply are not effective. In 2022, Toronto used an ADS to predict when high bacteria levels made it unsafe to swim at public beaches. Although the developer claimed the tool as 90 percent accurate, it fared far worse: “on 64 percent of the days when the water was unsafe, beaches remained open based on incorrect assessments.” Yet officials never overrode the recommendations produced by the tool.[31] Similarly, in 2017, a sepsis prediction tool that was deployed in hundreds of hospitals across the U.S. Despite having high accuracy when it was internally tested, a 2021 study found the tool missed two-third of the sepsis cases and led to a high rate of false alerts.[32]
As Princeton researchers Arvind Narayanan and Sayash Kapoor put it bluntly in AI Snake Oil:
“In contrast to generative AI, predictive AI often does not work at all.”[33] As described below, such tools are especially questionable when used to forecast individual human behavior.
5) “It’s tough to make predictions, especially about the future.”[34] Some phenomena, such as the movement of the planets, are easy to predict because they follow the laws of physics. Others, such as the weather, are chaotic systems that are harder to predict but can be modeled to make short-term predictions. Often, human behavior is even less predictable. Although some aggregate social patterns, such as traffic flow, are relatively consistent, how individuals will exercise their own agency cannot be predicted with certainty. How do we know if an individual will, for example, pay a debt? Even if they are fully committed to honoring the debt, unforeseen circumstances – illness or job loss – may prevent them from following through.
When it comes to predicting human behavior, ADS often fail to outperform simple statistical models. For instance, the Fragile Families challenge used over 10,000 data points about 4,000 children in 20 different U.S. cities to see how well AI could predict outcomes for specific children, such as their GPA. Complex AI models trained on all of this data fared no better than an analysis of a few key variables, such as the mother’s educational attainment.[35] Similarly, COMPAS, which used 137 data points to assess recidivism rates, fared no better than two data points about an individual – their age and number of prior offenses.[36]
A recently released study compiled a list of 47 applications of ADS that use machine learning to predict the future behavior or outcomes for individuals in eight domains: criminal justice, healthcare, welfare, finance, education, workplace, marketing, and recommender systems. The study concluded that such tools frequently fall well short of their purported benefits. The authors argue that developers and deployers of such systems should have the burden of demonstrating that their tools are not harmful.[37] As Narayanan and Kapoor write: “Accurately predicting people’s social behavior is not a solvable technology problem and determining people’s life chance on the basis of inherently faulty predictions will always be morally problematic.”[38]
Nevertheless, some AI companies market their products as capable of predicting future human behavior. For example, in 2019 RealPage announced its “AI Screening” tool as follows:
Traditional screening models use credit score, rent-to-income, debt-to-income and generic financial data to determine renter risk. While these factors broadly measure an applicant’s capability to pay financial obligations, including rent, RealPage developed industry-specific insights to determine the willingness to pay rent. Together, analyzing an applicant’s capability and willingness to pay rent is a superior risk assessment model to predict a renter’s financial performance.[39]
In 2023, the Federal Trade Commission warned companies that “we’re not yet living in the realm of science fiction, where computers can generally make trustworthy predictions of human behavior. Your performance claims would be deceptive if they lack scientific support or if they apply only to certain types of users or under certain conditions.”[40]
6) This bill creates a comprehensive transparency regime for ADS used in life-impacting decisions. Under the bill, an ADS is a computational process that produces “simplified outputs” – such as a score, recommendation, or classification – that are used to assist or replace human discretionary decisionmaking. The bill applies to a subset of ADS – “covered ADS” – that are used to make or facilitate a consequential decision. A consequential decision, in turn, is one that materially impacts the cost, terms, quality, or accessibility of any of several enumerated categories.
With respect to covered ADS, developers must conduct at least yearly performance evaluations that disclose the following:
- The expected accuracy and reliability of the covered ADS.
- Whether any disparate treatment is intended to occur, whether such treatment is necessary, and whether alternatives were considered.
- Whether the covered ADS is reasonably likely to result in disparate impacts, whether these impacts are necessary, and whether alternatives were considered.
- Whether measures were taken to mitigate the risk of unanticipated disparate impacts, including any unanticipated disparate impacts reported by deployers.
- Any reasonably foreseeable effects of fine tuning, as specified.
These evaluations are limited to “developer-approved uses” – the purposes for which the developer intends the ADS to be used. The developer is prohibited from representing that the ADS can perform in a manner not substantiated by the most recent performance evaluation conducted on the system. In subsequent performance evaluations, developers must consider feedback about the ADS’s performance from deployers. The developer must contract with independent third-party auditors to evaluate the developer’s compliance with their obligations related to performance evaluations. The developer must make a high-level summary of the auditor’s feedback publicly available. Developers must designate at least one employee to oversee compliance.
Deployers are required to provide subjects with pre-and post-decision notice that includes specified information about the covered ADS and the subject’s rights, as well as a reasonable opportunity to opt out of the use of the ADS. After the decision is made, subjects have 30 days to correct incorrect personal information used by the ADS or appeal the outcome. If the ADS affects more than 5,999 in a three-year period, deployers must, at the end of that period, contract with an independent third-party auditor to conduct an impact assessment. Unlike performance evaluations undertaken by developers, which are forward-looking evaluations of the model’s overall anticipated performance, impact assessments are a backwards-looking assessment of the ADS’s actual impact in the real world. Auditors performing impact assessments must document:
- Each developer-approved use of the covered ADS the deployer utilized.
- Differences in observed and expected accuracy and reliability.
- Whether disparate impacts resulted from the use of the ADS.
- Whether the deployer used the ADS outside the scope of a developer-approved use.
- Whether the deployer assumed the responsibilities of a developer by using the ADS, including by using the ADS outside the scope of developer-approved uses on more than 5,999 people over the course of the three-year period.
These impact assessments are provided to the deployer and the developer, who must take the results into account in future performance evaluations. A high-level summary of these impact assessments must be made publicly available. In certain circumstances, deployers that use ADS in specified ways assume the duties of a developer. Deployers must also designate at least one employee to oversee compliance.
To protect personal information, deployers would be required to minimize their collection, use, retention, and sharing of personal information used for consequential decisions. The bill also provides that it does not require the collection of information that is not otherwise collected in the ordinary course of business. Auditors of deployed systems may provide personal information of subjects to the developer of the system only if the subject first grants express consent.
The AG may obtain un-redacted performance evaluations or impact assessments, and share these with other enforcement entities as necessary. This documentation is exempted from Public Records Act requests. Public prosecutors, the Civil Rights Department, and Labor Commissioner may bring a civil action against a developer or deployer to bring an action for injunctive or declaratory relief, reasonable attorney’s fees and costs, and a civil penalty of up to $25,000 per violation. Developers and deployers who contract with third parties to perform duties under the bill are liable for the third parties’ failure to perform those duties. Auditors are also subject to liability under the bill.
The bill expressly provides that it does not override other rights, remedies or penalties under other provisions of law. The bill does not apply to ADS that solely serve cybersecurity functions or operate aircraft the national airspace. The bill also provides that compliance with the bill’s provisions is relevant to, but not conclusive of, anti-discrimination claims under the Unruh Civil Rights Act or FEHA.
7) Implementing the Blueprint for an AI Bill of Rights. The provisions of this bill are derived from the principles outlined in the Blueprint for an AI Bill of Rights, issued by the White House Office of Science and Technology Policy on October 4, 2022, and adopted by the Legislature in SCR 17 (Dodd, 2023). The Blueprint identifies five principles that should “guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence.”[41] The applicable principles, and the manner in which the bill implements them, are as follows:
Safe and effective systems. The Blueprint summarizes this principle as meaning that “you should be protected from unsafe or ineffective systems.”[42] This entails ongoing safety testing, including independent evaluation, and risk mitigation.[43] The bill would implement this principle by requiring developers to conduct forward-looking performance evaluations, subject to independent third-party audits. The bill would also require certain deployers of systems that affect 6,000 or more people to contract every three years with independent third-party auditors to conduct backward-looking impact assessments. Information would flow between developers and deployers to help identify and mitigate bias or ineffective or unsafe systems. Once completed, this documentation would be available to the Attorney General to review and, where necessary, initiate enforcement action. To ensure transparency and accountability, high-level summaries of audits must be made public.
Algorithmic discrimination protections. The Blueprint provides that “you should not face discrimination by algorithms[,] and systems should be used and designed in an equitable way.”[44] Accordingly, “Automated systems should be tested using a broad set of measures to assess whether the system components, both in pre-deployment testing and in-context deployment, produce disparities.”[45] The bill would implement this principle by requiring the testing and independent evaluation described above to determine whether the system is anticipated or found to have disparate impacts on those with protected characteristics. Developers must also disclose whether they took affirmative steps to mitigate potential disparate impacts. Because this creates a record of known potential discriminatory impacts that can be used in antidiscrimination claims, this creates a strong incentive to avoid releasing or using potentially discriminatory systems.
Data privacy. The Blueprint identifies data privacy as a “foundational and cross-cutting principle required for achieving all others in this framework.”[46] This bill implements this principle by requiring deployers to minimize their collection, use, retention, and sharing of personal information from subjects of consequential decisions. Auditors of deployed systems may provide personal information of subjects to the developer of the system only if the subject first grants express consent. Finally, the bill expressly provides that it does not mandate collection of information beyond that gathered in the ordinary course of business.
Notice and explanation. The Blueprint summarizes this principle as meaning that “you should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.”[47] The bill would implement this principle by requiring deployers, before finalizing the consequential decision, to provide notice to potential subjects with detailed information about the ADS. Post-decision notice must also be provided within 5 days. This notice gives subjects of consequential decisions notice and the ability to decide whether to exercise additional rights, discussed below.
Human alternatives, consideration, and fallback. The Blueprint summarizes this principle as meaning that “[y]ou should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.”[48] The bill would implement this principle by enabling subjects to opt out of the use of the covered ADS, correct their personal information used by the ADS, and appeal the decision made by the ADS.
The premises of this bill are, first, that Californians ought to be much more informed about the ADS being used to make consequential decisions in their lives, and second, that developers and deployers of ADS ought to be much more conscious and deliberate about what the impacts of their tools might be. In short, California ought to get a handle on ADS while it still can before they become yet another phenomenon – like social media, greenhouse gases, guns, and urban sprawl – that could have been dealt with deliberately and intelligently in the past, but was instead allowed to expand without any constraints past the point of being manageable. These protections are especially important in light of the federal government’s recent dismantling of programs that promote trustworthy AI as it has simultaneously demolished efforts to promote fairness and equality.
8) Comparison to AB 2930. This bill is substantially similar to the final version of AB 2930,[49] which was moved to the inactive file in the Senate. Key differences follow.
No “substantial factor.” AB 2930 applied to ADS that are a “substantial factor” in a consequential decision. “Substantial factor” was defined as “an element of a decisionmaking process that is capable of altering the outcome of the process.” This bill more straightforwardly applies to ADS that “make or facilitate” a consequential decision.
Shift from “algorithmic discrimination” to “disparate treatment” or “disparate impact.” Under AB 2930, developers and deployers were required to test for “algorithmic discrimination,” defined as “unlawful discrimination.” Rather than require businesses to formulate and disclose what amounts to a legal opinion about the risk of unlawful discrimination, this bill would instead focus on factual disclosures: whether the system is intended to treat protected classes differently, and whether testing shows that it will have a disparate impact on such classes. If developers and deployers use the system despite awareness of substantial and unjustified disparate impacts, they may be subject to a lawsuit on this basis under existing anti-discrimination laws. Certain federal and state laws allow for disparate impact claims; an exception is the Unruh Act, which applies to intentional discrimination only. Nevertheless, in Unruh Act cases, disparate impact “‘may be probative of intentional discrimination.’”[50] The bill does not change how anti-discrimination laws work; rather, it seeks to complement these laws through a process that creates a documentary record to help overcome the “black box” problem with many ADS.
“Developer-approved use.” This bill introduces the idea of “developer-approved uses” – a deployment context in which the developer intends an ADS to make or facilitate a consequential decision. Developer approved uses are important to the scope of the bill: they define the testing context for performance evaluations. They also limit how the developer may advertise the ADS’s capabilities. Finally, if a deployer uses the ADS in a manner that is not consistent with the developer-approved uses, the deployer may assume the developer’s responsibilities, ensuring that the developer is not on the hook for unintended uses of the ADS.
Audits. Several AI governance frameworks recommend third-party evaluation of AI tools.[51] The Blueprint states calls for third-party audits and states “entities should allow independent evaluation of potential algorithmic discrimination caused by automated systems they use or oversee.”[52] As discussed in this committee’s recent analysis of AB 1405 (Bauer-Kahan), which would create an enrollment process for AI auditors, auditing is a developing industry that promises to expand more jurisdictions require audits.[53] While AB 2930 did not address audits, this bill effectively requires two types. The first, applicable to developers, is a regulatory compliance audit to ensure they are properly carrying out their duties in conducting performance evaluations. The second, applicable to deployers of broadly used ADS, is an impact assessment that retrospectively evaluates the specific impact of the ADS in the deployment context.
Governance program. Whereas AB 2930 had a prescriptive oversight program for developers and deployers to oversee compliance with the bill’s obligations, this bill simply requires that developers and deployers designate an employee for this purpose and requires those employees to promptly review credible compliance issues.
Enforcement. The final version of AB 2930 allowed for administrative and civil enforcement by the Civil Rights Department, including a $25,000 civil penalty. Prior versions of the bill gave public prosecutors enforcement authority. This bill gives publics prosecutors as well as the Labor Commissioner civil enforcement. The civil penalty is up to $25,000. The bill does not provide for administrative enforcement.
9) Related efforts. A number of jurisdictions are taking action to regulate ADS. For instance, the European Union AI Act provides guardrails for what it deems “high-risk AI systems” to ensure transparency and fairness. Here in the United States, Senator Markey recently introduced his AI Civil Rights Act, which would impose stringent requirements on companies’ use of algorithms for consequential decisions.
New York City regulates the use of automated employment decision tools, requiring independent audits for bias and providing notice to job candidates.
A number of states have introduced legislation in this space, including New York, Connecticut, and Texas. The first comprehensive state-level regulation has come in Colorado. Signed into law May 17, 2024, that law places requirements on developers and deployers to use reasonable care to protect consumers from the risks of algorithmic discrimination. The bill has numerous commonalities with this bill and its predecessors.
The Virginia Legislature recently passed the High-Risk Artificial Intelligence Developer and Deployer Act, a comprehensive artificial intelligence bill focused on preventing algorithmic discrimination. However, on March 24, 2025, Virginia Governor Glenn Youngkin vetoed it.
Here in California, SB 420 (Padilla, 2025) would regulate the use of “high-risk” ADS. The bill contains several exceptions to its definition of ADS, exempts entities with 50 or fewer employees, allows state deployers to opt out of impact assessments, allows ADS to be used for two years without an impact assessment, does not enable subjects to opt out of the use of ADS, does not require audits, and provides a safe harbor to cure violations. It is supported by the Church State Council. A coalition of industry associations opposes.
SB 7 (McNerney, 2025) also regulates the use of ADS in the employment context, providing a minimum of 30 days’ notice to workers before deployment as well as post-deployment rights to notice, to correct information, and to appeal. It also restricts certain uses of ADS in the employment context. SB 7 appears broadly compatible with this bill.
At the administrative level, the California Privacy Protection Agency is engaged in an ongoing rulemaking on automated decisionmaking technology. This bill provides that deployers that are businesses subject to the Privacy’s agency’s jurisdiction may be subject to duly-adopted privacy-related regulations.
The Civil Rights Department recently finalized a rulemaking that clarifies how California antidiscrimination protections in the employment context apply to automated decision systems.
10) Opposition concerns. A coalition of industry trade organizations, led by California Chamber of Commerce, details its primary concerns:
The sweeping scope of businesses and industries captured, the inclusion of low-risk ADS applications, and establishment of new standards/grounds for discrimination, all have to be addressed to avoid creating a de-facto restraint on technology under the guise of an impact assessment bill. To start, businesses of 100 employees or less must be exempted. Fundamentally, both the scope and various key terms (in particular, “consequential decisions”, and “automated decision systems”, and “covered automated decision systems”, but also other terms such as “employment-related decisions”), all require additional clarity and/or narrowing. And insofar as the bill alters what constitutes discrimination, instead of ensuring that developers and deployers conduct evaluation and assessments that will promote responsible use of technologies to avoid violation of existing discrimination laws, the proposed change to the Unruh Civil Rights Act and introduction of new codified standards around “disparate impact” and “disparate treatment” should be fully stricken from the bill.
As to the size of the entities subject to the bill, small start-ups with a handful of employees may create widely-used ADS. Instead of focusing on the size of an entity, this bill’s requirement that deployers contract with auditors to perform impact assessments is limited to deployers whose ADS impacts more than 6,000 subjects over the course of a three-year period.
As to the definitions, these are largely in harmony with related legislative efforts described above. But the terms are broad and arguably encompass technologies that pose little threat of discrimination or that play a relatively minor role in the decision-making process. While it can be argued that it is better to err on the side of inclusion, it is essential to carefully calibrate the scope of the bill. The issue warrants continued discussion with stakeholders.
Finally, as to the Unruh Act, the bill provides that a developer or deployer’s compliance with the bill is relevant to, but not conclusive of, such actions. This is an evidentiary truism meant to clarify that the bill does not alter how existing anti-discrimination laws work. Rather, the bill complements these laws with a rigorous testing process that provides incentives for harms to be proactively mitigated and creates a documentary record that can be used in potential lawsuits.
The coalition continues:
Untenable opt-out and pre-and post-decision notice obligations must be deleted. All opt out and notice obligations, including the right to appeal which will have drastically different impact depending on context, must be deleted in full to make the bill workable. Such requirements are not only largely unworkable in many contexts, but also wholly unrelated to and unnecessary for there to be a bill that would require evaluations/assessments that to help reduce bias and discriminatory outcomes from the development and deployment of such ADS.
The bill recognizes that notice and opt-out provisions may be unworkable in certain situations, such as a medical emergency. Additional exceptions may need to be added. But it is not clear why these are categorically unworkable and must be deleted in full. A right to appeal, on the other hand, raises more substantial workability questions. In many cases, it appears that it is not feasible to rectify a decision, such as when a job was given to another person as a result of a discriminatory ADS. That said, the bill allows for the deployer to deny a request for appeal as long as they provide an explanation of the basis for the denial to the subject appealing the decision.
The coalition continues:
Third-party auditing requirements should be deleted in favor of self-assessments, particularly when third-party auditing would effectively grant a monopoly today and creates a costly cottage industry tomorrow, with access to highly sensitive and proprietary information, and no standards of care or liability. The third-party auditor requirement imposes excessive and unnecessary costs on businesses, increasing IP risks and operational inefficiencies, without providing any added consumer protections. In fact, the requirement will have the opposite effect by driving up costs for consumers. With a limited number of auditors available, a legal mandate would create a surge in demand, allowing existing auditors to charge inflated fees without competition. As businesses absorb these new compliance costs, they will be forced to raise prices, ultimately burdening consumers, reducing sales, and hindering economic growth in California. Furthermore, this requirement exceeds the scope of other U.S. laws and proposals, making California the most expensive jurisdiction for compliance. Given such impacts, the third-party auditor requirement for both developers and deployers must be removed to make this bill remotely viable.
As described above, AI governance programs such as the Blueprint generally call for independent third-party evaluation rather than simply entrusting entities to self-regulate. Opponents are right that an audit can be costly; hence audits are not required for deployers who use systems that impact less than 6,000 subjects over the course of a three year period. The timing of audits and scope of entities subject to those audits is an issue that warrants continued discussion with stakeholders.
A single enforcer is necessary to promote consistency and economic stability. Statutory penalties are far too heavy handed for a violation that involves no actual harm. Allowing multiple enforcement entities will invariably create confusion for compliance, whereas the provision of a single enforcement entity (the Attorney General) will promote consistent interpretation and application across the state. Whereas the penalties in AB 2930 were $25,000 per violation for algorithmic discrimination only, the current penalty structure in the bill could feasibly have devastating impacts on businesses, big and most certainly small. Specifically, it allows for courts to grant plaintiffs a civil penalty of up to $25,000 per violation, without adequate clarity as to what constitutes each violation.
This bill enables civil enforcement by the Attorney General, local public prosecutors, the Civil Rights Department, and the Labor Commissioner with respect to employment-related issues. Notably, the bill, unlike AB 2930, does not provide for administrative enforcement. Further clarifications as to what constitutes a violation may be in order.
Finally, the coalition adds:
Preemption protections are needed, and current regulatory activities must be addressed. Preemption is needed both to prevent conflict with localities and, similarly, to prevent concerns around state departments and agencies over-regulating this technology and getting ahead of the Legislature and Governor. These issues are too important to Californians across the state and our struggling economy to significantly delegate to unelected officials.
As local governments and administrative agencies consider their own approach to this issue, there is the looming possibility of multiple overlapping sets of ADS regulations. There is virtue in uniformity, provided that the uniform standard is sufficiently strong. The author may wish to continue exploring this issue.
ARGUMENTS IN SUPPORT: A broad coalition of civil society and labor organizations jointly write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile, to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones. Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives. Specifically, the legislation:
Requires these tools to be tested before they are used on the public: Requires that people who make and use these tools test them to make sure they do not create harm and comply with our existing rights to non-discrimination before they are sold and used on the public. It ensures that these tests are verified by an independent third party.
Provides a notice to people that this tool will be used to make a critical decision about their life: Provides people the information they need to understand where these tools are showing up in their lives and how they’ll be used to determine their housing, healthcare, and job outcomes.
Provides an explanation to people who were subject to a decision made with these tools: Provides people an explanation of what the tool did, what personal information it used about them to make the decision and what role the tool played in making the decision.
Ensures that every day people have more control over how these tools are used in their daily lives: Through this bill, people will have the right to opt out of the use of an ADS tool in a critical decision about them; they will be able to correct information that the tool used to make the decision if it is inaccurate; and they will have the right to appeal the decision.
Pre-deployment testing protects Californians by ensuring that businesses understand the risks and accuracy of the systems they’re purchasing and deploying to make consequential decisions about Californians. The strong notice and explanation provisions in this bill are a critical first step for consumers to know how these tools are being used and to be able to exercise their right to correct their data or appeal a decision. These protections are more important than ever as we see a Federal pullback from safety guardrails on AI and as other states and jurisdictions pass laws recognizing the danger inherent in letting these systems enter into our lives without clear protections and safeguards for the public.
ARGUMENTS IN OPPOSITION: A coalition of industry trade organizations, led by California Chamber of Commerce, writes that the bill:
. . .fails to focus on the stated objective of requirements on high-risk automated decision systems (ADS) to the detriment of every industry in the state of California, and from small business to large. Instead, the bill broadly targets businesses of all sizes, across every industry, and regulates even lower risk applications of ADS, including those that are already in use, and unnecessarily alters what constitutes discrimination under California law. Such overreach not only exposes smaller businesses to significant – if not devastating – liability even for mere errors that caused no harm to consumers, but it would also hinder many beneficial uses of ADS, including but not limited to: enabling faster approvals and expanded access to credit; enhancing real-time fraud detection; fostering job creation and new industries; improving efficiency to help level the playing field between small and large businesses; addressing major societal challenges such as bias and discrimination, economic inequality, climate change, injustices in the criminal system, disaster relief, and humanitarian aid; and advancing new treatments for previously incurable diseases.
To be clear, whether decisions are being made by humans from start to end, or a byproduct of using or incorporating new technologies in the decision-making process, we take our responsibility to not discriminate with the utmost seriousness. We believe that algorithmic discrimination, or discrimination that results from AI-enabled technologies, is already prohibited under our anti-discrimination laws because our laws are rights-based and not technology-specific. We also agree that companies need to take care to reduce bias and discrimination in decisions that have legal impact on the provision or denial of fundamental rights or essential opportunities and services, which is why so many of them already conduct impact assessments.
But it would be incredibly short-sighted for regulation to stifle innovation when alternative (human-decision driven) systems may be equally, if not more, flawed, and when properly developed and deployed ADS can enhance fairness and accountability. While ADS may pose unique challenges in terms of bias, they also pose unique advantages to combatting it as well: for example, ADS decision-making processes can be more transparent and traceable than the exercise of human discretion. The structured nature of these tools offer opportunities for detection and correction that can be more challenging to come by in human decision-making processes where human bias is often more subtle and harder to detect. Unfortunately, we believe AB 1018 will have an undesired chilling effect on the technology and make it that much harder to develop the very tools that can help combat bias in decisionmaking.
As introduced, AB 1018 goes far beyond ensuring developers and deployers of these technologies act responsibly to adhere to existing anti-discrimination protections and veers into an indirect restriction upon the usage of the technology itself, discouraging technological innovation and usage by making it so onerous and risky that businesses are realistically pushed back toward the alternative, human driven process – which we know based on historical evidence will not help eliminate bias and discrimination in any way.
REGISTERED SUPPORT / OPPOSITION:
Support
California State Council of Service Employees International Union (seiu California) (Co-Sponsor)
Techequity Action (Co-Sponsor)
Acce Action (alliance of Californians for Community Empowerment)
American Federation of Musicians Local 47
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Employment Lawyers Association
California Federation of Labor Unions, Afl-cio
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, Aft, Afl-cio
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center (EPIC)
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Justice2jobs Coalition
Kapor Center
LA Defensa
Laane (los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Northern California District Council of the International Longshore and Warehouse Union (ILWU)
Oakland Privacy
Orange County Employees Association
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Transparency Coalition.ai
Udw/afscme Local 3930
Ufcw - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
Opposition
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
Association of California Life & Health Insurance Companies
Association of National Advertisers
California Chamber of Commerce
Chamber of Progress
Civil Justice Association of California
Civil Justice Association of California (CJAC)
College Board
Computer and Communications Industry Association
Consumer Data Industry Association
Independent Insurance Agents & Brokers of California, INC.
Internet.works
National Association of Mutual Insurance Companies
Pacific Association of Domestic Insurance Companies
Personal Insurance Federation of California
Public Risk Innovation, Solutions, and Management (PRISM)
Security Industry Association
Silicon Valley Leadership Group
Software Information Industry Association
Techca
Technet
Oppose Unless Amended
Advanced Medical Technology Association (ADVAMED)
America's Physician Groups
American College of Obstetricians & Gynecologists - District Ix
Calbroadband
Calcom Association
California Association of Health Plans
California Bankers Association
California Community Banking Network
California Credit Union League
California Hospital Association
California Life Sciences
California Medical Association (CMA)
California Mortgage Bankers Association
California Radiological Society
CTIA
Kaiser Permanente
Mortgage Bankers Association
Securities Industry and Financial Markets Association
Sutter Health
Analysis Prepared by: Josh Tosney / P. & C.P. / (916) 319-2200
The White House, “Blueprint for an AI Bill of Rights,” (Oct. 2022), p. 14, https://bidenwhitehouse.archives.gov/ostp/ai-bill-of-rights/ (“Blueprint”). Despite the use of the term “AI” in its title, the Blueprint focuses on ADS. ↑
IBM, What is machine learning? www.ibm.com/topics/machine-learning. ↑
Neil Savage, “Breaking into the black box of artificial intelligence,” Nature (March 29, 2022). ↑
Brian Christian, “The Alignment Problem: Machine Learning and Human Values” (Norton 2020, 1st ed.), p. 85. ↑
OpenAI, How ChatGPT and Our Language Models Are Developed,https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-foundation-models-are-developed. ↑
Netflix, How Netflix’s Recommendations System Works, https://help.netflix.com/en/node/100639. ↑
Hany Farid, “Artificial Intelligence: A Primer for Legal Practitioners” Artificial Intelligence: Legal Issues, Policy, and Practical Strategies (American Bar Association, 2024), p. 73. ↑
Gov. Code § 11546.45.5(a)(1). ↑
See e.g. Santariano & Metz, “Using A.I. to Detect Breast Cancer That Doctors Miss,” New York Times (Mar. 5, 2023), https://www.nytimes.com/2023/03/05/technology/artificial-intelligence-breast-cancer-detection.html. ↑
Rohit Sehgal, “AI Needs Data More Than Data Needs AI,” Forbes (Oct. 5, 2023), https://www.forbes.com/sites/forbestechcouncil/2023/10/05/ai-needs-data-more-than-data-needs-ai/. ↑
Arvind Narayanan and Sayash Kapoor, AI Snake Oil: What Artificial Intelligence Can Do, Can’t Do, and How to Tell the Difference (1st ed. 2024), pp. 52-53. ↑
Emmanuel Martinez and Lauren Kirchner, “The Secret Bias Hidden in Mortgage-Approval Algorithms,” Markup (Aug. 25, 2021), https://themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms. ↑
“When Algorithms Give Real Students Imaginary Grades,” New York Times (Sept. 8, 2020), https://www.nytimes.com/2020/09/08/opinion/international-baccalaureate-algorithm-grades.html. ↑
Id. ↑
Aditya Malik, “AI Bias In Recruitment: Ethical Implications And Transparency,” Forbes (Sep. 25, 2023), https://www.forbes.com/sites/forbestechcouncil/2023/09/25/ai-bias-in-recruitment-ethical-implications-and-transparency/. ↑
Jeffrey Dastin, “Amazon scraps secret AI recruiting tool that showed bias against women,” Reuters (Oct. 9, 2018), https://www.reuters.com/article/amazoncom-jobs-automation/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSL2N1VB1FQ/. ↑
Blueprint, supra, p. 17. ↑
Caleb J Colón-Rodríguez, “Shedding Light on Healthcare Algorithmic and Artificial Intelligence Bias,” U.S. Department of Health & Human Services Office of Minority Health, (Jul. 12, 2023), https://pmc.ncbi.nlm.nih.gov/articles/PMC6875681/. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” p. 461, https://oag.ca.gov/system/files/media/full-ca-reparations.pdf. ↑
Obermeyer et al, “Dissecting racial bias in an algorithm used to manage the health of populations,” Science 2019., 366(6464):447–453. ↑
Erin Smith and Helen Vogell, “How Your Shadow Credit Score Could Decide Whether You Get an Apartment” ProPublica (Mar. 29, 2022), https://www.propublica.org/article/how-your-shadow-credit-score-could-decide-whether-you-get-an-apartment. While the mere compilation and provision of background information is not subject to this bill, processing that information through an algorithm to produce a simplified output such as a score or yes/no recommendation for a prospective tenant – thereby outsourcing human discretionary decisionmaking – generally falls under this bill. ↑
“Attorney General Bonta Submits Comment Letter Recommending Reforms to the Tenant Screening Process” (May 31, 2023), https://oag.ca.gov/news/press-releases/attorney-general-bonta-submits-comment-letter-recommending-reforms-tenant#:~:text=OAKLAND. ↑
Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, “Machine Bias,” Propublica, (May 23, 2016), https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” supra, p. 420. ↑
See Mathews v. Eldridge (1976) 424 U.S. 319, 333; Daniel Keats Citron, Technological Due Process (2007) 85 Wash. U. L. Rev. 1249, 1249 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1012360 ↑
Boracas et al., Fairness and Machine Learning: Limitations and Opportunities (2023) pp. 213-214. ↑
Brian Christian, “The Alignment Problem: Machine Learning and Human Values” (Norton 2020, First Ed.), pp. 82-84. ↑
See Hilke Schellmann, “The Algorithm: How AI Decides Who Gets Hired, Monitored, Promoted, and Fired and Why We Need to Fight Back Now” (1st ed. 2024). ↑
Snake Oil, supra, at p. 73. ↑
Id. at p. 51. ↑
Id. at p. 50. ↑
Wong et al, “External valiation of a widely implemented proprietary sepsis prediction model in hospitalized patients” JAMA Int. Med. 181 (Aug. 2021), 1065–1070, https://doi.org/10.1001/jamainternmed.2021.2626. ↑
Snake Oil, supra, at p. 9. ↑
Quote from Yogi Berra. ↑
Snake Oil, supra, at p. 73. ↑
Id. at p. 80. ↑
Angelina Wang et al. 2023. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA: ACM, 2023), 626–26. ↑
Snake Oil, 15. ↑
“RealPage Release AI Screening,” (Jun. 26, 2019), https://www.realpage.com/news/realpage-releases-ai-screening/. Emphasis in original. ↑
Snake Oil, supra, 25. ↑
Blueprint, supra, p. 14. ↑
Id. at p. 15. ↑
Ibid. ↑
Id. at p. 23. ↑
Id. at p. 27. ↑
Id. at p. 31. ↑
Id. at p. 40. ↑
Id. at p. 46. ↑
That version was, however, specific to employment, whereas prior versions applied to a broader set of contexts. ↑
Liapes v. Facebook, Inc. (2023) 95 Cal.App.5th 910, 926. ↑
See Selinger et al, “AI Audits: Who, When, How...Or Even If?” (Sep. 2023), in Collaborative Intelligence: How Humans and AI are Transforming our World, MIT Press (Forthcoming), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4568208. ↑
Blueprint, supra, p. 28; see also p. 19 (third-party auditors to demonstrate safety and effectiveness of system). ↑
E. Mulvaney, “NYC Targets Artificial Intelligence Bias in Hiring Under New Law,” Bloomberg Law, 2021, https://news.bloomberglaw.com/daily-labor-report/nyc-tar gets-artificial-intelligence-bias-in-hiring-under-new-law. ↑
Date of Hearing: April 22, 2025
Fiscal: Yes
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
ABPCA Bill Id:AB 1018 (
Author:Bauer-Kahan) – As Amended Ver:April 10, 2025
SUBJECT: Automated decision systems
SYNOPSIS
Automated Decision Systems (ADS) typically use artificial intelligence (AI) that produce simplified outputs – such as scores, classifications, or recommendations – to assist or replace human discretionary decisionmaking. Often these decisions have low-stakes; sometimes, however, they can meaningfully impact the public’s rights, opportunities, or access to critical resources or services. ADS can process enormous datasets, identify hidden patterns, and make decisions with efficiency and scale that vastly exceeds human capabilities. But relying on ADS to make consequential decisions can be hazardous if the systems are not trained carefully or tested thoroughly: the datasets they are trained on are often unrepresentative or contaminated with bias, the inferences they draw from those datasets are often inscrutable, and these systems can fail to accurately account for the complexity of human behavior. Without human-centered oversight, particularly in consequential contexts such as employment, housing, healthcare, and criminal justice, these impacts can be irreparable.
In 2022, the White House Office of Science and Technology Policy released the Blueprint for an AI Bill of Rights, which identifies five principles that should “guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence.”[1] These principles would (1) protect the public from unsafe and ineffective ADS, (2) prevent algorithmic discrimination, (3) protect data privacy, (4) give subjects notice and explanation when ADS are used, and (5) provide for human alternatives, consideration, and fallback. The Blueprint calls for rigorous testing, monitoring, and independent evaluation of ADS to ensure that ADS are at least as fair and trustworthy as the humans they are displacing.
This bill is the author’s third attempt to realize that vision. In broad strokes, the bill is similar to its predecessors: it seeks to create a comprehensive transparency regime for developers and deployers of ADS that are used for consequential decisionmaking. The author’s goal is to prevent algorithmic discrimination on the basis of protected characteristics and to ensure that ADS can be trusted to perform reliably and accurately. The bill would require deployers of ADS to give subjects of consequential decisions notice, an ability to opt out of the use of the ADS, a chance to correct any personal information used by the ADS to make the decision, and a right to appeal the outcome of the decision. The bill also provides for independent evaluation by third-party auditors – a substantial but controversial provision not found in the prior bills. Enforcement of the bill may be undertaken by the Attorney General, public prosecutors, the Civil Rights Department, or Labor Commissioner, within their respective jurisdictions.
The bill is co-sponsored by SEIU California and TechEquity and is supported by a broad coalition of civil society, consumer protection, and labor organizations. Supporters argue that the bill establishes common-sense guardrails to prevent inaccurate or discriminatory decisions by ADS – protections that are especially needed after the federal government’s retreat from AI regulation.
The bill is opposed by a coalition of industry trade organizations, led by California Chamber of Commerce, who set forth several concerns relating to the bill’s breadth, workability, and costs. A number of other businesses and organizations take an oppose-unless-amended position.
If passed by this Committee, this bill will next be heard by the Judiciary Committee.
THIS BILL:
- Defines key terms, including:
- “Automated decision system,” or ADS, means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is designed or used to assist or replace human discretionary decisionmaking and materially impacts natural persons. The term does not include a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
- “Consequential decision” means a decision that materially impacts the cost, terms, quality, or accessibility of any of the following to a natural person:
- Employment-related decisions, as defined.
- Education and vocational training as they relate to specified categories.
- Housing and lodging as they relate to rentals or short-term housing and lodging, home appraisals, rental subsidies, and publicly supported housing.
- Essential utilities: electricity, heat, water, transportation, and municipal trash and sewage services.
- Family planning, adoption services, reproductive services, and assessments related to child protective services.
- Health care and health insurance.
- Financial services.
- The criminal justices system with respect to pretrial release, sentencing, and alternatives to incarceration.
- Legal services, private arbitrations, and mediation.
- Elections, as they relate to specified categories.
- Access to government benefits or services or assignment of penalties by a government entity.
- Places of public accommodation, as specified.
- Insurance.
- Internet and telecommunications access.
- “Covered ADS” means an ADS that is designed or used to make or facilitate a consequential decision.
- “Deployer” means a person, partnership, state or local government agency, corporation, or developer that uses a covered ADS to make or facilitate a consequential decision, either directly or by contracting with a third party for that purpose.
- “Developer” means a person, partnership, state or local government agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an automated decision system that makes or facilitates a consequential decision, either directly or by contracting with a third party for those purposes.
- “Developer-approved use” means a deployment context in which a developer intends a covered ADS to make or facilitate a consequential decision, and includes any reasonably foreseeable fine tuning of the covered ADS.
- “Disparate impact” means a differential effect on a group of individuals who share a protected characteristic.
- “Disparate treatment” means differential treatment of an individual or group of individuals on the basis of a protected characteristic.
- “Express consent” means affirmative written authorization granted in response to a notice that meets specified requirements.
- “Fine-tune” means to adjust the model parameters of an ADS through exposure to additional data.
- “Personal information” has the same meaning as defined in the California Consumer Privacy Act.
- “Protected characteristic” means a characteristic listed in the Unruh Civil Rights Act.
- “Substantial modification” means a new version, release, update, or other modification to a covered ADS that materially changes its uses or outputs, but does not include modifications resulting from fine-tuning.
- “Trade secret” has the same meaning as provided under the Uniform Trade Secrets Act.
- Requires developers to:
- Conduct initial performance evaluations on ADS either before they are first deployed or, if they were deployed before January 1, 2026, by January 1, 2027.
- Conduct subsequent performance evaluations on ADS no more than a year apart, including following any substantial modification or fine-tuning.
- In conducting a performance evaluation, for each developer-approved use, document:
- The expected accuracy and reliability of the covered ADS.
- Whether any disparate treatment is intended to occur, whether such treatment is necessary, and whether alternatives were considered.
- Whether the covered ADS is reasonably likely to result in disparate impacts, whether these impacts are necessary, and whether alternatives were considered.
- Whether measures were taken to mitigate the risk of unanticipated disparate impacts, including any unanticipated disparate impacts reported by deployers.
- Any reasonably foreseeable effects of fine tuning, as specified.
- Contract with an independent third-party auditor to assess the developer’s compliance with requirements for performance evaluations. Provide the auditor with necessary documentation, which may be redacted to protect trade secrets. Consider and attempt to incorporate auditor feedback into subsequent versions of the ADS. Make a high-level summary of the feedback publicly available.
- Disclose to deployers results of the most recent performance evaluation, instructions for using and fine tuning the ADS, an explanation of circumstances in which a deployer’s fine tuning of the ADS requires them to assume the responsibilities of a developer, and any technical information needed to comply with the bill.
- Provide specified information received from audits of performance evaluation to deployers, including discrepancies from anticipated performance, unanticipated disparate impacts, and steps the deployer can take to mitigate discrepancies.
- Consider and attempt to incorporate feedback from auditors into development of subsequent versions of covered ADS.
- Ensure documentation provided to deployers is provided in a clear manner in a language commonly used with the developers, and that such documentation is retained for as long as the ADS remains deployed or available to potential deployers plus 10 years.
- Refrain from representing that an ADS can perform in a manner not substantiated by the results of the most recent performance evaluation, which under the bill constitutes false advertising.
- Designate at least one employee to oversee compliance with the bill’s requirements.
- As of January 1, 2027, requires deployers to:
- Before finalizing a consequential decision, provide subjects of the decision:
- Written notice with information about the covered ADS, including the personal characteristics it measures or assesses, sources of personal information collected from the subject, structure and format of the outputs, whether a natural person reviews the outputs or outcome before finalizing the consequential decision, and the subject’s right to opt out.
- A reasonable opportunity to opt out of the use of the ADS to make or facilitate the consequential decision.
- A deployer need not comply with (i) or (ii) if the subject is having a medical emergency or the deployer is subject to the federal Gramm-Leach-Bliley Act, which governs financial institutions.
- After finalizing a consequential decision, provide the subjects of the decision:
- Another written notice similar to the one described above, within 5 days.
- An opportunity to, within 30 days, correct incorrect personal information or appeal the outcome, as specified. If the exercise of either right would result in a different outcome, the deployer must rectify the outcome within 30 days of making this determination. If not, deployers must inform the subject. Deployers may deny a request to correct information but must provide the subject with an explanation for the basis of the denial and, for requests to correct the information, allow them a reasonable opportunity to request deletion of their personal information.
- Minimize their collection, use, retention, and sharing of personal information from subjects of consequential decisions.
- If the ADS impacts more than 5,999 people in a three-year period, contract with a third-party independent auditor to conduct an impact assessment on the covered ADS before January 1, 2030 and every three years thereafter.
- Assume the responsibilities of the developer if the deployer uses a covered ADS that impacts 5,999 or more people in a given three-year period and does not receive documentation relating to the developer-approved uses and performance evaluations for the covered ADS, or substantially modifies the ADS and uses it to impact 5,999 people in a three year period or otherwise makes the modified system available to potential deployers.
- Retain specified documentation for as long as the ADS is deployed plus 10 years.
- Designate at least one employee to oversee compliance with the bill.
- Comply with any additional requirements related to covered ADS promulgated by the California Privacy Protection Agency in duly adopted regulations.
- Before finalizing a consequential decision, provide subjects of the decision:
- Requires auditors conducting impact assessments every three years on behalf of deployers who use covered ADS that impact more than 5,999 people in that period to:
- Document: each developer-approved use of the covered ADS the deployer utilized; differences in observed and expected accuracy and reliability; whether disparate impacts resulted from the use of the ADS; whether the deployer used the ADS outside the scope of a developer-approved use; and whether the deployer assumed the responsibilities of a developer.
- Provide the results of the impact assessment to the developer and deployer.
- Make a high-level summary of the results of the impact assessment publicly available.
- Enables the Attorney General to obtain an unredacted performance evaluation or impact assessment, and to share that documentation with other enforcement entities as necessary for enforcement purposes. Such documentation is exempt from the California Public Records Act.
- Enables public prosecutors, the Civil Rights Department, and Labor Commissioner (with respect to employment-related violations) to bring a civil action against a developer or deployer to bring an action for injunctive or declaratory relief, reasonable attorney’s fees and costs, and a civil penalty of up to $25,000 per violation.
- Provides that a developer or deployer who contracts with a third party to comply with duties required under the bill, other than duties relating to auditing, is liable for the third party’s failure to comply with those duties.
- Does not:
- Apply to ADS that solely serve a cybersecurity function or operate aircraft in the national airspace.
- Apply to use of a consumer credit score.
- Limit rights, remedies, and penalties under any other law.
- Authorize use of an ADS, or disparate impacts and treatments, otherwise limited, restricted, or prohibited under any other law.
- Provides that in an action alleging a violation of the Unruh Civil Rights Act or FEHA in which an ADS is alleged to have committed or facilitated the violation, the defendant’s compliance with this bill is relevant to, but not conclusive of, the claim.
EXISTING LAW:
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act. (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act, which provides that all persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever. (Civil Code § 51(b).)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
- Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
- Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civ. Code § 3426.1(d).)
- Establishes the California Privacy Protection Agency (Privacy Agency) and vests it with full administrative power, authority, and jurisdiction to implement and enforce the California Consumer Privacy Act (CPPA) of 2018. (Civ. Code § 1798.199.10.)
- Requires the California Privacy Protection Agency to issue regulations governing access and opt-out rights with respect to businesses’ use of automated decisionmaking technology. (Civ. Code § 1798.185.)
- Establishes prohibitions on false and deceptive advertising. (Bus. & Prof. Code § 17500 et seq.)
COMMENTS:
1) Author’s statement. According to the author:
The use of automated decision systems (ADS) has become prevalent among Californian’s daily lives and used within various sectors – including, housing, employment, and even in criminal justice sentencing and probation decisions. However, the algorithms that power ADS are often vulnerable to issues such as unrepresentative data, faulty classifications, and flawed design. These shortcomings can result in biased, discriminatory, or unfair outcomes. Rather than solving systemic problems, poorly designed ADS can worsen the very harms they aim to address—ultimately hurting the people they are meant to help. AB 1018 provides the necessary guardrails by regulating the development and deployment of an ADS used to make consequential decisions. Specifically, it requires developers to conduct an initial performance evaluation of an ADS by January 1, 2027. Additionally, the current role that an ADS plays in a consequential decision is hidden from consumers. Fundamentally, consumers should have the right to be well informed of how these ADS are used in life altering decisions. AB 1018 calls for transparency on the usage of ADS by requiring notice to consumers by deployers before and after a consequential decision. It is crucial that we take the necessary steps to ensure the technology is used responsibly and can be trusted. Well-intentioned but flawed technology is a matter of state concern. Guardrails and accountability are needed to ensure that technology does not further marginalize communities or broaden inequities.
2) Automated Decision Systems. As a general matter, ADS use AI to make life-impacting decisions. AI refers to the mimicking of human intelligence by artificial systems such as computers. AB 2885 (Bauer-Kahan, Stats. 2024, Ch. 843) defined the term as “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.” AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; unlike other computer functions, however, AI is able to accomplish tasks that are normally performed by humans.
Most modern AI tools are created through a process known as “machine learning.” Whereas traditional programming is manually written by a programmer and can only follow pre-defined rules, with machine learning algorithms, programmers do not determine the rules the algorithm will follow. Instead, the AI is allowed to automatically explore the structure of data and infer the relationship between the data and desired outputs.[2] Machine-learning algorithms are often criticized for being “black boxes” that generate predictions and outcomes that cannot be clearly explained.[3] “It’s often observed in the field that the most powerful models are on the whole the least intelligible, and the most intelligible are among the least accurate.”[4]
The process of exposing a naïve AI to data is known as “training.” The algorithm that an AI develops during training is known as its “model.” Training is the secret sauce of machine learning. At its core, training is an optimization problem wherein a model attempts to identify specific parameters – “weights” – that minimize the difference between predicted outcomes and actual outcomes. How an input is transformed into an output depends on the specific algorithm that is developed by a model. Once trained, AI can be used to process new, never-before-seen data.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from generative AI (GenAI,) which are trained on massive datasets in order to produce detailed text, images, audio, and video. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that is trained on the written contents of the internet.[5] When Netflix suggests content to a viewer, its recommendation is produced by predictive AI that is trained on the viewing habits of Netflix users.[6]
ADS typically use predictive AI to generate outputs that influence real-world outcomes.[7] Existing law defines an ADS to include “a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.”[8]
When the stakes of these outcomes are high, such as predicting a person’s risk of defaulting on a loan or committing a crime, the use of an ADS can cut both ways: these systems can create efficiencies, ensure uniformity, and detect intricate patterns among massive datasets, leading to incredibly beneficial applications and breakthroughs.[9] On the other hand, use of ADS can also raise significant questions of fairness, accountability, and transparency. Government Code section 11546.45.5(a)(4) defines “high-risk” ADS as those that are “used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.”
This bill incorporates the existing definition of ADS and applies to any such system used to make or facilitate a “consequential decision,” defined under the bill as a decision that materially impacts the cost, terms, quality, or accessibility of several enumerated categories with respect to a natural person.
3) Algorithmic discrimination. There is a well-known saying in computer science: “garbage in, garbage out.” The performance of an ADS is directly impacted by the quality, quantity, and relevance of the data used to train it.[10] If the data used to train the ADS is biased, the tool’s outputs will be similarly biased. Over the past thirty years, several industries have been forced to contend with this fact as they have attempted to introduce ADS into their workflows. Specific examples follow.
Child welfare: In 2016, Allegheny County, Pennsylvania adopted a family screening tool to predict which families should be investigated by social workers for possible removal of maltreated children. The tool was trained only on data from poorer families who used public services such as Medicaid. Because its training dataset lacked examples of wealthier families, the tool disproportionately targeted poorer families.[11]
Credit and loan approval. Financial tools that utilize ADS are similarly susceptible to bias and discrimination. An investigation by The Markup (and co-published by the Associated Press) revealed that in 2019, lenders were more likely to deny home loans to people of color; in particular, lenders were 40 percent more likely to turn down Latino applicants for loans, 50 percent more likely to deny Asian/Pacific Islander applicants, and 70 percent more likely to deny Native American applicants than similar white applicants. Lenders were 80 percent more likely to reject Black applicants than similar white applicants. In every case, the prospective borrowers of color looked almost exactly the same on paper as the white applicants, except for their race. According to the report, mortgage-approval algorithms play a major role in perpetuating these inequities.[12]
Education. Professor Broussard’s article in the New York Times details how “the International Baccalaureate—a global program that awards prestigious diplomas to high school students—canceled its usual in-person exams because of the [COVID-19] pandemic” and instead “used an algorithm to ‘predict’ student grades based on an array of student information, including teacher-estimated grades and past performance by students in each school.”[13] Tens of thousands of students, surprised to find out they failed, protested the results. “High-achieving, low-income students were hit particularly hard: many took the exams expecting to earn college credit with their scores and save thousands of dollars on tuition.”[14]
Employment. It is no secret that people of various races, genders, and cultures are not distributed equally throughout the workforce. An ADS that is trained on historical data to make hiring decisions will be predisposed to maintain the ratios it is trained on; as described by Aditya Malik, the Founder and CEO of Valuematrix.ai:
Generative AI, for all its grandeur, has the potential to perpetuate latent biases inherited from human creators. A disconcerting echo of historical prejudices may inadvertently seep into the algorithms. Imagine a scenario where previous senior managers, driven by biases of gender, age, faith or race, rejected candidates for misguided reasons. The AI, if not vigilantly curated, might misconstrue these patterns as indicators of incompetence, thus exacerbating the exclusion of qualified candidates from underrepresented backgrounds.[15]
This was notoriously experienced by Amazon, who considered automating their hiring practices in the early 2010s. They opted against this approach in 2015 when they realized that their ADS-enabled system was not rating candidates in a gender-neutral way. In fact, their system was excluding women from the pool of acceptable candidates because it had been trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of inequities across the tech industry.[16]
ADS are being employed in the workforce as well. For example, “[a] company installed AI-powered cameras in its delivery vans in order to evaluate the road safety habits of its drivers, but the system incorrectly penalized drivers when other cars cut them off or when other events beyond their control took place on the road. As a result, drivers were incorrectly ineligible to receive a bonus.”[17]
Healthcare. When ADS are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. In 2007, an ADS was developed to help doctors estimate whether it was safe for people who had delivered previous children through cesarean section to deliver subsequent children vaginally – a risky procedure. The ADS considered various health relevant factors as it made its decision, such as the woman’s age, her reason for the previous cesarean, and how long ago the cesarean had been performed. However, a 2017 study found that the ADS was biased; it predicted Black and Latino people were less likely to have a successful vaginal birth after a cesarean than non-Hispanic white women. As a result, doctors performed more cesareans on Black and Latino people than on white people.[18] These discrepancies perpetuate historical biases – Black Americans, for example, have historically received a lower standard of healthcare than their white counterparts.[19]
Similarly, a 2019 study found strong racial bias in a system used to identify patients with a high risk of adverse health outcomes. The ADS assigned Black patients lower likelihoods than equally at-risk White patients. The authors found that this happened because the ADS was designed to predict healthcare costs instead of needs. Because the healthcare system has historically spent less on care for Black patients than White patients for the same health conditions, the ADS was, in essence, issuing a prediction that mirrored and perpetuated past discrimination.[20]
Housing. A recent ProPublica article found that tenant screening companies compile information on renters beyond credit reports – including criminal background, evictions filings, medical debt, and student loans – and then use algorithms that “try to predict how risky it is to rent to a potential tenant based on characteristics they share with other tenants” and “assign applicants scores or provide landlords a yes-or-no recommendation.”[21] In 2023, concerns about these tools led several Attorneys General – California’s Rob Bonta included – to write in a comment letter to the Federal Trade Commission and the Consumer Financial Protection Bureau:
Landlords and tenant screening companies increasingly rely on problematic screening algorithms that combine various information to generate a single score or result indicating how “safe” it would be to rent to a prospective tenant. These algorithms lack transparency and can be inaccurate, and can have a discriminatory impact on underserved communities. The attorneys general recommend, among other things, requiring that tenant screening companies disclose their reliance on algorithms and screen algorithm models for bias against protected classes and prohibiting the use of certain types of records in screening reports.[22]
Sentencing and bail decisions. ADS are frequently used to inform sentencing and bail decisions. These tools are trained using historical data, and the predictions they make can therefore reflect historical bias. A 2016 ProPublica study dove into the use of one such tool – COMPAS – in Broward County, Florida.[23] The study determined that Black defendants were far more likely than white defendants to be incorrectly judged to be at a higher risk of recidivism, while white defendants were more likely than Black defendants to be incorrectly flagged as low risk.
The for-profit company that developed this tool, Northpointe, does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. These discrepancies mirror historical injustices perpetuated against Black Americans by California’s criminal justice system.[24] The use of a proprietary algorithm by government actors also raises significant due process questions, as it becomes difficult for individuals to understand, let alone challenge, arbitrary government deprivations of life, liberty, or property.[25]
Structural effects of discriminatory ADS: ADS becoming more pervasive across institutions raises the risk of a feedback loop that compounds and reinforces discrimination. As Borocas et al. write in Fairness and Machine Learning, “predictive systems have the effect of transferring advantages from one phase of life to the next, and one generation to the next.” This can compound and amplify injustices: “individuals are subject to a series of decisions over the course of their lives, and the effects of these decisions both accumulate and compound over time. When a person receives (or is denied) one opportunity, they are likely to appear more (or less) qualified at their next encounter with a predictive system.”[26]
4) Unsafe or ineffective systems. In addition to discriminatory outcomes, some ADS are unsafe or simply ineffective regardless of who the subjects of the prediction are. A few common types of flawed ADS are described below.
Spurious correlations. Accurate predictions may nevertheless lead to bad decisions. In one example, a hospital trained AI models on a dataset of 15,000 pneumonia patients in order to develop a model that could identify which pneumonia patients were at the greatest risk in order to triage new patients. During testing, it was discovered that one of the most accurate models recommended outpatient status for asthmatics – a life-threateningly dangerous error based on an accurate statistical correlation: asthmatics are less likely to die from pneumonia than the general population precisely because asthma is such a serious risk factor that asthmatics automatically get elevated care.[27]
In other cases, the correlations that machine-learning ADS may rely on have little to do with the attributes they purport to measure. Some hiring tools record videos of applicants responding to pre-recorded questions and attempt to assess their fitness for a job based on their facial expression and intonation. Such tools can be gamed by making simple changes to the subject’s appearance, such as wearing glasses, or the background of the room, such as adding more books on the bookshelf – leading to increased scores. Journalist Hilke Schellmann found she was able to obtain consistently high scores on one hiring ADS despite responding by reading an irrelevant Wikipedia entry in German.[28]
Irrelevant datasets. “AI reflects its training data. It learns patterns about the people who make up the data, and the decisions made by AI reflect these patterns. But when the decision subjects come from a population with different characteristics than those in the training data, the model’s decisions are likely to be wrong.”[29] For instance, the Ohio Risk Assessment System was trained on data from just 452 defendants from Ohio, but has been deployed in several other states, despite its small and unrepresentative dataset.[30]
Snake oil. Some tools, although marketed as automating precision, are simply are not effective. In 2022, Toronto used an ADS to predict when high bacteria levels made it unsafe to swim at public beaches. Although the developer claimed the tool as 90 percent accurate, it fared far worse: “on 64 percent of the days when the water was unsafe, beaches remained open based on incorrect assessments.” Yet officials never overrode the recommendations produced by the tool.[31] Similarly, in 2017, a sepsis prediction tool that was deployed in hundreds of hospitals across the U.S. Despite having high accuracy when it was internally tested, a 2021 study found the tool missed two-third of the sepsis cases and led to a high rate of false alerts.[32]
As Princeton researchers Arvind Narayanan and Sayash Kapoor put it bluntly in AI Snake Oil:
“In contrast to generative AI, predictive AI often does not work at all.”[33] As described below, such tools are especially questionable when used to forecast individual human behavior.
5) “It’s tough to make predictions, especially about the future.”[34] Some phenomena, such as the movement of the planets, are easy to predict because they follow the laws of physics. Others, such as the weather, are chaotic systems that are harder to predict but can be modeled to make short-term predictions. Often, human behavior is even less predictable. Although some aggregate social patterns, such as traffic flow, are relatively consistent, how individuals will exercise their own agency cannot be predicted with certainty. How do we know if an individual will, for example, pay a debt? Even if they are fully committed to honoring the debt, unforeseen circumstances – illness or job loss – may prevent them from following through.
When it comes to predicting human behavior, ADS often fail to outperform simple statistical models. For instance, the Fragile Families challenge used over 10,000 data points about 4,000 children in 20 different U.S. cities to see how well AI could predict outcomes for specific children, such as their GPA. Complex AI models trained on all of this data fared no better than an analysis of a few key variables, such as the mother’s educational attainment.[35] Similarly, COMPAS, which used 137 data points to assess recidivism rates, fared no better than two data points about an individual – their age and number of prior offenses.[36]
A recently released study compiled a list of 47 applications of ADS that use machine learning to predict the future behavior or outcomes for individuals in eight domains: criminal justice, healthcare, welfare, finance, education, workplace, marketing, and recommender systems. The study concluded that such tools frequently fall well short of their purported benefits. The authors argue that developers and deployers of such systems should have the burden of demonstrating that their tools are not harmful.[37] As Narayanan and Kapoor write: “Accurately predicting people’s social behavior is not a solvable technology problem and determining people’s life chance on the basis of inherently faulty predictions will always be morally problematic.”[38]
Nevertheless, some AI companies market their products as capable of predicting future human behavior. For example, in 2019 RealPage announced its “AI Screening” tool as follows:
Traditional screening models use credit score, rent-to-income, debt-to-income and generic financial data to determine renter risk. While these factors broadly measure an applicant’s capability to pay financial obligations, including rent, RealPage developed industry-specific insights to determine the willingness to pay rent. Together, analyzing an applicant’s capability and willingness to pay rent is a superior risk assessment model to predict a renter’s financial performance.[39]
In 2023, the Federal Trade Commission warned companies that “we’re not yet living in the realm of science fiction, where computers can generally make trustworthy predictions of human behavior. Your performance claims would be deceptive if they lack scientific support or if they apply only to certain types of users or under certain conditions.”[40]
6) This bill creates a comprehensive transparency regime for ADS used in life-impacting decisions. Under the bill, an ADS is a computational process that produces “simplified outputs” – such as a score, recommendation, or classification – that are used to assist or replace human discretionary decisionmaking. The bill applies to a subset of ADS – “covered ADS” – that are used to make or facilitate a consequential decision. A consequential decision, in turn, is one that materially impacts the cost, terms, quality, or accessibility of any of several enumerated categories.
With respect to covered ADS, developers must conduct at least yearly performance evaluations that disclose the following:
- The expected accuracy and reliability of the covered ADS.
- Whether any disparate treatment is intended to occur, whether such treatment is necessary, and whether alternatives were considered.
- Whether the covered ADS is reasonably likely to result in disparate impacts, whether these impacts are necessary, and whether alternatives were considered.
- Whether measures were taken to mitigate the risk of unanticipated disparate impacts, including any unanticipated disparate impacts reported by deployers.
- Any reasonably foreseeable effects of fine tuning, as specified.
These evaluations are limited to “developer-approved uses” – the purposes for which the developer intends the ADS to be used. The developer is prohibited from representing that the ADS can perform in a manner not substantiated by the most recent performance evaluation conducted on the system. In subsequent performance evaluations, developers must consider feedback about the ADS’s performance from deployers. The developer must contract with independent third-party auditors to evaluate the developer’s compliance with their obligations related to performance evaluations. The developer must make a high-level summary of the auditor’s feedback publicly available. Developers must designate at least one employee to oversee compliance.
Deployers are required to provide subjects with pre-and post-decision notice that includes specified information about the covered ADS and the subject’s rights, as well as a reasonable opportunity to opt out of the use of the ADS. After the decision is made, subjects have 30 days to correct incorrect personal information used by the ADS or appeal the outcome. If the ADS affects more than 5,999 in a three-year period, deployers must, at the end of that period, contract with an independent third-party auditor to conduct an impact assessment. Unlike performance evaluations undertaken by developers, which are forward-looking evaluations of the model’s overall anticipated performance, impact assessments are a backwards-looking assessment of the ADS’s actual impact in the real world. Auditors performing impact assessments must document:
- Each developer-approved use of the covered ADS the deployer utilized.
- Differences in observed and expected accuracy and reliability.
- Whether disparate impacts resulted from the use of the ADS.
- Whether the deployer used the ADS outside the scope of a developer-approved use.
- Whether the deployer assumed the responsibilities of a developer by using the ADS, including by using the ADS outside the scope of developer-approved uses on more than 5,999 people over the course of the three-year period.
These impact assessments are provided to the deployer and the developer, who must take the results into account in future performance evaluations. A high-level summary of these impact assessments must be made publicly available. In certain circumstances, deployers that use ADS in specified ways assume the duties of a developer. Deployers must also designate at least one employee to oversee compliance.
To protect personal information, deployers would be required to minimize their collection, use, retention, and sharing of personal information used for consequential decisions. The bill also provides that it does not require the collection of information that is not otherwise collected in the ordinary course of business. Auditors of deployed systems may provide personal information of subjects to the developer of the system only if the subject first grants express consent.
The AG may obtain un-redacted performance evaluations or impact assessments, and share these with other enforcement entities as necessary. This documentation is exempted from Public Records Act requests. Public prosecutors, the Civil Rights Department, and Labor Commissioner may bring a civil action against a developer or deployer to bring an action for injunctive or declaratory relief, reasonable attorney’s fees and costs, and a civil penalty of up to $25,000 per violation. Developers and deployers who contract with third parties to perform duties under the bill are liable for the third parties’ failure to perform those duties. Auditors are also subject to liability under the bill.
The bill expressly provides that it does not override other rights, remedies or penalties under other provisions of law. The bill does not apply to ADS that solely serve cybersecurity functions or operate aircraft the national airspace. The bill also provides that compliance with the bill’s provisions is relevant to, but not conclusive of, anti-discrimination claims under the Unruh Civil Rights Act or FEHA.
7) Implementing the Blueprint for an AI Bill of Rights. The provisions of this bill are derived from the principles outlined in the Blueprint for an AI Bill of Rights, issued by the White House Office of Science and Technology Policy on October 4, 2022, and adopted by the Legislature in SCR 17 (Dodd, 2023). The Blueprint identifies five principles that should “guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence.”[41] The applicable principles, and the manner in which the bill implements them, are as follows:
Safe and effective systems. The Blueprint summarizes this principle as meaning that “you should be protected from unsafe or ineffective systems.”[42] This entails ongoing safety testing, including independent evaluation, and risk mitigation.[43] The bill would implement this principle by requiring developers to conduct forward-looking performance evaluations, subject to independent third-party audits. The bill would also require certain deployers of systems that affect 6,000 or more people to contract every three years with independent third-party auditors to conduct backward-looking impact assessments. Information would flow between developers and deployers to help identify and mitigate bias or ineffective or unsafe systems. Once completed, this documentation would be available to the Attorney General to review and, where necessary, initiate enforcement action. To ensure transparency and accountability, high-level summaries of audits must be made public.
Algorithmic discrimination protections. The Blueprint provides that “you should not face discrimination by algorithms[,] and systems should be used and designed in an equitable way.”[44] Accordingly, “Automated systems should be tested using a broad set of measures to assess whether the system components, both in pre-deployment testing and in-context deployment, produce disparities.”[45] The bill would implement this principle by requiring the testing and independent evaluation described above to determine whether the system is anticipated or found to have disparate impacts on those with protected characteristics. Developers must also disclose whether they took affirmative steps to mitigate potential disparate impacts. Because this creates a record of known potential discriminatory impacts that can be used in antidiscrimination claims, this creates a strong incentive to avoid releasing or using potentially discriminatory systems.
Data privacy. The Blueprint identifies data privacy as a “foundational and cross-cutting principle required for achieving all others in this framework.”[46] This bill implements this principle by requiring deployers to minimize their collection, use, retention, and sharing of personal information from subjects of consequential decisions. Auditors of deployed systems may provide personal information of subjects to the developer of the system only if the subject first grants express consent. Finally, the bill expressly provides that it does not mandate collection of information beyond that gathered in the ordinary course of business.
Notice and explanation. The Blueprint summarizes this principle as meaning that “you should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.”[47] The bill would implement this principle by requiring deployers, before finalizing the consequential decision, to provide notice to potential subjects with detailed information about the ADS. Post-decision notice must also be provided within 5 days. This notice gives subjects of consequential decisions notice and the ability to decide whether to exercise additional rights, discussed below.
Human alternatives, consideration, and fallback. The Blueprint summarizes this principle as meaning that “[y]ou should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.”[48] The bill would implement this principle by enabling subjects to opt out of the use of the covered ADS, correct their personal information used by the ADS, and appeal the decision made by the ADS.
The premises of this bill are, first, that Californians ought to be much more informed about the ADS being used to make consequential decisions in their lives, and second, that developers and deployers of ADS ought to be much more conscious and deliberate about what the impacts of their tools might be. In short, California ought to get a handle on ADS while it still can before they become yet another phenomenon – like social media, greenhouse gases, guns, and urban sprawl – that could have been dealt with deliberately and intelligently in the past, but was instead allowed to expand without any constraints past the point of being manageable. These protections are especially important in light of the federal government’s recent dismantling of programs that promote trustworthy AI as it has simultaneously demolished efforts to promote fairness and equality.
8) Comparison to AB 2930. This bill is substantially similar to the final version of AB 2930,[49] which was moved to the inactive file in the Senate. Key differences follow.
No “substantial factor.” AB 2930 applied to ADS that are a “substantial factor” in a consequential decision. “Substantial factor” was defined as “an element of a decisionmaking process that is capable of altering the outcome of the process.” This bill more straightforwardly applies to ADS that “make or facilitate” a consequential decision.
Shift from “algorithmic discrimination” to “disparate treatment” or “disparate impact.” Under AB 2930, developers and deployers were required to test for “algorithmic discrimination,” defined as “unlawful discrimination.” Rather than require businesses to formulate and disclose what amounts to a legal opinion about the risk of unlawful discrimination, this bill would instead focus on factual disclosures: whether the system is intended to treat protected classes differently, and whether testing shows that it will have a disparate impact on such classes. If developers and deployers use the system despite awareness of substantial and unjustified disparate impacts, they may be subject to a lawsuit on this basis under existing anti-discrimination laws. Certain federal and state laws allow for disparate impact claims; an exception is the Unruh Act, which applies to intentional discrimination only. Nevertheless, in Unruh Act cases, disparate impact “‘may be probative of intentional discrimination.’”[50] The bill does not change how anti-discrimination laws work; rather, it seeks to complement these laws through a process that creates a documentary record to help overcome the “black box” problem with many ADS.
“Developer-approved use.” This bill introduces the idea of “developer-approved uses” – a deployment context in which the developer intends an ADS to make or facilitate a consequential decision. Developer approved uses are important to the scope of the bill: they define the testing context for performance evaluations. They also limit how the developer may advertise the ADS’s capabilities. Finally, if a deployer uses the ADS in a manner that is not consistent with the developer-approved uses, the deployer may assume the developer’s responsibilities, ensuring that the developer is not on the hook for unintended uses of the ADS.
Audits. Several AI governance frameworks recommend third-party evaluation of AI tools.[51] The Blueprint states calls for third-party audits and states “entities should allow independent evaluation of potential algorithmic discrimination caused by automated systems they use or oversee.”[52] As discussed in this committee’s recent analysis of AB 1405 (Bauer-Kahan), which would create an enrollment process for AI auditors, auditing is a developing industry that promises to expand more jurisdictions require audits.[53] While AB 2930 did not address audits, this bill effectively requires two types. The first, applicable to developers, is a regulatory compliance audit to ensure they are properly carrying out their duties in conducting performance evaluations. The second, applicable to deployers of broadly used ADS, is an impact assessment that retrospectively evaluates the specific impact of the ADS in the deployment context.
Governance program. Whereas AB 2930 had a prescriptive oversight program for developers and deployers to oversee compliance with the bill’s obligations, this bill simply requires that developers and deployers designate an employee for this purpose and requires those employees to promptly review credible compliance issues.
Enforcement. The final version of AB 2930 allowed for administrative and civil enforcement by the Civil Rights Department, including a $25,000 civil penalty. Prior versions of the bill gave public prosecutors enforcement authority. This bill gives publics prosecutors as well as the Labor Commissioner civil enforcement. The civil penalty is up to $25,000. The bill does not provide for administrative enforcement.
9) Related efforts. A number of jurisdictions are taking action to regulate ADS. For instance, the European Union AI Act provides guardrails for what it deems “high-risk AI systems” to ensure transparency and fairness. Here in the United States, Senator Markey recently introduced his AI Civil Rights Act, which would impose stringent requirements on companies’ use of algorithms for consequential decisions.
New York City regulates the use of automated employment decision tools, requiring independent audits for bias and providing notice to job candidates.
A number of states have introduced legislation in this space, including New York, Connecticut, and Texas. The first comprehensive state-level regulation has come in Colorado. Signed into law May 17, 2024, that law places requirements on developers and deployers to use reasonable care to protect consumers from the risks of algorithmic discrimination. The bill has numerous commonalities with this bill and its predecessors.
The Virginia Legislature recently passed the High-Risk Artificial Intelligence Developer and Deployer Act, a comprehensive artificial intelligence bill focused on preventing algorithmic discrimination. However, on March 24, 2025, Virginia Governor Glenn Youngkin vetoed it.
Here in California, SB 420 (Padilla, 2025) would regulate the use of “high-risk” ADS. The bill contains several exceptions to its definition of ADS, exempts entities with 50 or fewer employees, allows state deployers to opt out of impact assessments, allows ADS to be used for two years without an impact assessment, does not enable subjects to opt out of the use of ADS, does not require audits, and provides a safe harbor to cure violations. It is supported by the Church State Council. A coalition of industry associations opposes.
SB 7 (McNerney, 2025) also regulates the use of ADS in the employment context, providing a minimum of 30 days’ notice to workers before deployment as well as post-deployment rights to notice, to correct information, and to appeal. It also restricts certain uses of ADS in the employment context. SB 7 appears broadly compatible with this bill.
At the administrative level, the California Privacy Protection Agency is engaged in an ongoing rulemaking on automated decisionmaking technology. This bill provides that deployers that are businesses subject to the Privacy’s agency’s jurisdiction may be subject to duly-adopted privacy-related regulations.
The Civil Rights Department recently finalized a rulemaking that clarifies how California antidiscrimination protections in the employment context apply to automated decision systems.
10) Opposition concerns. A coalition of industry trade organizations, led by California Chamber of Commerce, details its primary concerns:
The sweeping scope of businesses and industries captured, the inclusion of low-risk ADS applications, and establishment of new standards/grounds for discrimination, all have to be addressed to avoid creating a de-facto restraint on technology under the guise of an impact assessment bill. To start, businesses of 100 employees or less must be exempted. Fundamentally, both the scope and various key terms (in particular, “consequential decisions”, and “automated decision systems”, and “covered automated decision systems”, but also other terms such as “employment-related decisions”), all require additional clarity and/or narrowing. And insofar as the bill alters what constitutes discrimination, instead of ensuring that developers and deployers conduct evaluation and assessments that will promote responsible use of technologies to avoid violation of existing discrimination laws, the proposed change to the Unruh Civil Rights Act and introduction of new codified standards around “disparate impact” and “disparate treatment” should be fully stricken from the bill.
As to the size of the entities subject to the bill, small start-ups with a handful of employees may create widely-used ADS. Instead of focusing on the size of an entity, this bill’s requirement that deployers contract with auditors to perform impact assessments is limited to deployers whose ADS impacts more than 6,000 subjects over the course of a three-year period.
As to the definitions, these are largely in harmony with related legislative efforts described above. But the terms are broad and arguably encompass technologies that pose little threat of discrimination or that play a relatively minor role in the decision-making process. While it can be argued that it is better to err on the side of inclusion, it is essential to carefully calibrate the scope of the bill. The issue warrants continued discussion with stakeholders.
Finally, as to the Unruh Act, the bill provides that a developer or deployer’s compliance with the bill is relevant to, but not conclusive of, such actions. This is an evidentiary truism meant to clarify that the bill does not alter how existing anti-discrimination laws work. Rather, the bill complements these laws with a rigorous testing process that provides incentives for harms to be proactively mitigated and creates a documentary record that can be used in potential lawsuits.
The coalition continues:
Untenable opt-out and pre-and post-decision notice obligations must be deleted. All opt out and notice obligations, including the right to appeal which will have drastically different impact depending on context, must be deleted in full to make the bill workable. Such requirements are not only largely unworkable in many contexts, but also wholly unrelated to and unnecessary for there to be a bill that would require evaluations/assessments that to help reduce bias and discriminatory outcomes from the development and deployment of such ADS.
The bill recognizes that notice and opt-out provisions may be unworkable in certain situations, such as a medical emergency. Additional exceptions may need to be added. But it is not clear why these are categorically unworkable and must be deleted in full. A right to appeal, on the other hand, raises more substantial workability questions. In many cases, it appears that it is not feasible to rectify a decision, such as when a job was given to another person as a result of a discriminatory ADS. That said, the bill allows for the deployer to deny a request for appeal as long as they provide an explanation of the basis for the denial to the subject appealing the decision.
The coalition continues:
Third-party auditing requirements should be deleted in favor of self-assessments, particularly when third-party auditing would effectively grant a monopoly today and creates a costly cottage industry tomorrow, with access to highly sensitive and proprietary information, and no standards of care or liability. The third-party auditor requirement imposes excessive and unnecessary costs on businesses, increasing IP risks and operational inefficiencies, without providing any added consumer protections. In fact, the requirement will have the opposite effect by driving up costs for consumers. With a limited number of auditors available, a legal mandate would create a surge in demand, allowing existing auditors to charge inflated fees without competition. As businesses absorb these new compliance costs, they will be forced to raise prices, ultimately burdening consumers, reducing sales, and hindering economic growth in California. Furthermore, this requirement exceeds the scope of other U.S. laws and proposals, making California the most expensive jurisdiction for compliance. Given such impacts, the third-party auditor requirement for both developers and deployers must be removed to make this bill remotely viable.
As described above, AI governance programs such as the Blueprint generally call for independent third-party evaluation rather than simply entrusting entities to self-regulate. Opponents are right that an audit can be costly; hence audits are not required for deployers who use systems that impact less than 6,000 subjects over the course of a three year period. The timing of audits and scope of entities subject to those audits is an issue that warrants continued discussion with stakeholders.
A single enforcer is necessary to promote consistency and economic stability. Statutory penalties are far too heavy handed for a violation that involves no actual harm. Allowing multiple enforcement entities will invariably create confusion for compliance, whereas the provision of a single enforcement entity (the Attorney General) will promote consistent interpretation and application across the state. Whereas the penalties in AB 2930 were $25,000 per violation for algorithmic discrimination only, the current penalty structure in the bill could feasibly have devastating impacts on businesses, big and most certainly small. Specifically, it allows for courts to grant plaintiffs a civil penalty of up to $25,000 per violation, without adequate clarity as to what constitutes each violation.
This bill enables civil enforcement by the Attorney General, local public prosecutors, the Civil Rights Department, and the Labor Commissioner with respect to employment-related issues. Notably, the bill, unlike AB 2930, does not provide for administrative enforcement. Further clarifications as to what constitutes a violation may be in order.
Finally, the coalition adds:
Preemption protections are needed, and current regulatory activities must be addressed. Preemption is needed both to prevent conflict with localities and, similarly, to prevent concerns around state departments and agencies over-regulating this technology and getting ahead of the Legislature and Governor. These issues are too important to Californians across the state and our struggling economy to significantly delegate to unelected officials.
As local governments and administrative agencies consider their own approach to this issue, there is the looming possibility of multiple overlapping sets of ADS regulations. There is virtue in uniformity, provided that the uniform standard is sufficiently strong. The author may wish to continue exploring this issue.
ARGUMENTS IN SUPPORT: A broad coalition of civil society and labor organizations jointly write:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADS have long existed in our communities. ADS have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, healthcare, employment, credit, and many other critical services Californians need. The potential harms and biases of these systems have been well documented—from banks using lending models that were twice as likely to deny Black applicants compared to White applicants with the same financial profile, to healthcare providers using ADS that significantly underestimated the healthcare needs of Black patients compared to White ones. Technology should not be a pass to violate our civil and labor rights, this legislation regulates the misuse of automated systems.
AB 1018 would enact common-sense guardrails to help ensure that developers and deployers of these tools are testing for discriminatory outcomes prior to utilizing the tool and ensuring that consumers have the information they need to understand the role that an ADS is playing in critical decisions and what rights they have when these systems impact critical areas of their lives. Specifically, the legislation:
Requires these tools to be tested before they are used on the public: Requires that people who make and use these tools test them to make sure they do not create harm and comply with our existing rights to non-discrimination before they are sold and used on the public. It ensures that these tests are verified by an independent third party.
Provides a notice to people that this tool will be used to make a critical decision about their life: Provides people the information they need to understand where these tools are showing up in their lives and how they’ll be used to determine their housing, healthcare, and job outcomes.
Provides an explanation to people who were subject to a decision made with these tools: Provides people an explanation of what the tool did, what personal information it used about them to make the decision and what role the tool played in making the decision.
Ensures that every day people have more control over how these tools are used in their daily lives: Through this bill, people will have the right to opt out of the use of an ADS tool in a critical decision about them; they will be able to correct information that the tool used to make the decision if it is inaccurate; and they will have the right to appeal the decision.
Pre-deployment testing protects Californians by ensuring that businesses understand the risks and accuracy of the systems they’re purchasing and deploying to make consequential decisions about Californians. The strong notice and explanation provisions in this bill are a critical first step for consumers to know how these tools are being used and to be able to exercise their right to correct their data or appeal a decision. These protections are more important than ever as we see a Federal pullback from safety guardrails on AI and as other states and jurisdictions pass laws recognizing the danger inherent in letting these systems enter into our lives without clear protections and safeguards for the public.
ARGUMENTS IN OPPOSITION: A coalition of industry trade organizations, led by California Chamber of Commerce, writes that the bill:
. . .fails to focus on the stated objective of requirements on high-risk automated decision systems (ADS) to the detriment of every industry in the state of California, and from small business to large. Instead, the bill broadly targets businesses of all sizes, across every industry, and regulates even lower risk applications of ADS, including those that are already in use, and unnecessarily alters what constitutes discrimination under California law. Such overreach not only exposes smaller businesses to significant – if not devastating – liability even for mere errors that caused no harm to consumers, but it would also hinder many beneficial uses of ADS, including but not limited to: enabling faster approvals and expanded access to credit; enhancing real-time fraud detection; fostering job creation and new industries; improving efficiency to help level the playing field between small and large businesses; addressing major societal challenges such as bias and discrimination, economic inequality, climate change, injustices in the criminal system, disaster relief, and humanitarian aid; and advancing new treatments for previously incurable diseases.
To be clear, whether decisions are being made by humans from start to end, or a byproduct of using or incorporating new technologies in the decision-making process, we take our responsibility to not discriminate with the utmost seriousness. We believe that algorithmic discrimination, or discrimination that results from AI-enabled technologies, is already prohibited under our anti-discrimination laws because our laws are rights-based and not technology-specific. We also agree that companies need to take care to reduce bias and discrimination in decisions that have legal impact on the provision or denial of fundamental rights or essential opportunities and services, which is why so many of them already conduct impact assessments.
But it would be incredibly short-sighted for regulation to stifle innovation when alternative (human-decision driven) systems may be equally, if not more, flawed, and when properly developed and deployed ADS can enhance fairness and accountability. While ADS may pose unique challenges in terms of bias, they also pose unique advantages to combatting it as well: for example, ADS decision-making processes can be more transparent and traceable than the exercise of human discretion. The structured nature of these tools offer opportunities for detection and correction that can be more challenging to come by in human decision-making processes where human bias is often more subtle and harder to detect. Unfortunately, we believe AB 1018 will have an undesired chilling effect on the technology and make it that much harder to develop the very tools that can help combat bias in decisionmaking.
As introduced, AB 1018 goes far beyond ensuring developers and deployers of these technologies act responsibly to adhere to existing anti-discrimination protections and veers into an indirect restriction upon the usage of the technology itself, discouraging technological innovation and usage by making it so onerous and risky that businesses are realistically pushed back toward the alternative, human driven process – which we know based on historical evidence will not help eliminate bias and discrimination in any way.
REGISTERED SUPPORT / OPPOSITION:
Support
California State Council of Service Employees International Union (seiu California) (Co-Sponsor)
Techequity Action (Co-Sponsor)
Acce Action (alliance of Californians for Community Empowerment)
American Federation of Musicians Local 47
American Federation of Musicians, Local 7
Asian Americans Advancing Justice Southern California
California Center for Movement Legal Services
California Employment Lawyers Association
California Federation of Labor Unions, Afl-cio
California Immigrant Policy Center
California Initiative on Technology and Democracy
California National Organization for Women
California Professional Firefighters
California School Employees Association
California Women's Law Center
CFT – a Union of Educators & Classified Professionals, Aft, Afl-cio
Citizen’s Privacy Coalition
Common Sense Media
Consumer Attorneys of California
Consumer Federation of America
Consumer Federation of California
Consumer Reports
Courage California
East Bay Community Law Center
Economic Security California Action
Electronic Privacy Information Center (EPIC)
End Child Poverty California Powered by Grace
Equal Rights Advocates
Fair Housing Advocates of Northern California
Felony Murder Elimination Project
Greenlining Institute
Justice2jobs Coalition
Kapor Center
LA Defensa
Laane (los Angeles Alliance for a New Economy)
Lawyers Committee for Civil Rights of the San Francisco Bay Area
Northern California District Council of the International Longshore and Warehouse Union (ILWU)
Oakland Privacy
Orange County Employees Association
Parent Voices
Powerswitch Action
Privacy Rights Clearinghouse
Public Citizen
Restoring Hope California
Rubicon Programs
Smart Justice California
Surveillance Resistance Lab
Transparency Coalition.ai
Udw/afscme Local 3930
Ufcw - Western States Council
Vision Y Compromiso
Women's Foundation California
Working Partnerships USA
Worksafe
Opposition
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
Association of California Life & Health Insurance Companies
Association of National Advertisers
California Chamber of Commerce
Chamber of Progress
Civil Justice Association of California (CJAC)
College Board
Computer and Communications Industry Association
Consumer Data Industry Association
Independent Insurance Agents & Brokers of California, INC.
Internet.works
National Association of Mutual Insurance Companies
Pacific Association of Domestic Insurance Companies
Personal Insurance Federation of California
Public Risk Innovation, Solutions, and Management (PRISM)
Security Industry Association
Silicon Valley Leadership Group
Software Information Industry Association
Techca
Technet
Oppose Unless Amended
Advanced Medical Technology Association (ADVAMED)
America's Physician Groups
American College of Obstetricians & Gynecologists - District Ix
Calbroadband
Calcom Association
California Association of Health Plans
California Bankers Association
California Community Banking Network
California Credit Union League
California Hospital Association
California Life Sciences
California Medical Association (CMA)
California Mortgage Bankers Association
California Radiological Society
CTIA
Kaiser Permanente
Mortgage Bankers Association
Securities Industry and Financial Markets Association
Sutter Health
Analysis Prepared by: Josh Tosney / P. & C.P. / (916) 319-2200
The White House, “Blueprint for an AI Bill of Rights,” (Oct. 2022), p. 14, https://bidenwhitehouse.archives.gov/ostp/ai-bill-of-rights/ (“Blueprint”). Despite the use of the term “AI” in its title, the Blueprint focuses on ADS. ↑
IBM, What is machine learning? www.ibm.com/topics/machine-learning. ↑
Neil Savage, “Breaking into the black box of artificial intelligence,” Nature (March 29, 2022). ↑
Brian Christian, “The Alignment Problem: Machine Learning and Human Values” (Norton 2020, 1st ed.), p. 85. ↑
OpenAI, How ChatGPT and Our Language Models Are Developed,https://help.openai.com/en/articles/7842364-how-chatgpt-and-our-foundation-models-are-developed. ↑
Netflix, How Netflix’s Recommendations System Works, https://help.netflix.com/en/node/100639. ↑
Hany Farid, “Artificial Intelligence: A Primer for Legal Practitioners” Artificial Intelligence: Legal Issues, Policy, and Practical Strategies (American Bar Association, 2024), p. 73. ↑
Gov. Code § 11546.45.5(a)(1). ↑
See e.g. Santariano & Metz, “Using A.I. to Detect Breast Cancer That Doctors Miss,” New York Times (Mar. 5, 2023), https://www.nytimes.com/2023/03/05/technology/artificial-intelligence-breast-cancer-detection.html. ↑
Rohit Sehgal, “AI Needs Data More Than Data Needs AI,” Forbes (Oct. 5, 2023), https://www.forbes.com/sites/forbestechcouncil/2023/10/05/ai-needs-data-more-than-data-needs-ai/. ↑
Arvind Narayanan and Sayash Kapoor, AI Snake Oil: What Artificial Intelligence Can Do, Can’t Do, and How to Tell the Difference (1st ed. 2024), pp. 52-53. ↑
Emmanuel Martinez and Lauren Kirchner, “The Secret Bias Hidden in Mortgage-Approval Algorithms,” Markup (Aug. 25, 2021), https://themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms. ↑
“When Algorithms Give Real Students Imaginary Grades,” New York Times (Sept. 8, 2020), https://www.nytimes.com/2020/09/08/opinion/international-baccalaureate-algorithm-grades.html. ↑
Id. ↑
Aditya Malik, “AI Bias In Recruitment: Ethical Implications And Transparency,” Forbes (Sep. 25, 2023), https://www.forbes.com/sites/forbestechcouncil/2023/09/25/ai-bias-in-recruitment-ethical-implications-and-transparency/. ↑
Jeffrey Dastin, “Amazon scraps secret AI recruiting tool that showed bias against women,” Reuters (Oct. 9, 2018), https://www.reuters.com/article/amazoncom-jobs-automation/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSL2N1VB1FQ/. ↑
Blueprint, supra, p. 17. ↑
Caleb J Colón-Rodríguez, “Shedding Light on Healthcare Algorithmic and Artificial Intelligence Bias,” U.S. Department of Health & Human Services Office of Minority Health, (Jul. 12, 2023), https://pmc.ncbi.nlm.nih.gov/articles/PMC6875681/. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” p. 461, https://oag.ca.gov/system/files/media/full-ca-reparations.pdf. ↑
Obermeyer et al, “Dissecting racial bias in an algorithm used to manage the health of populations,” Science 2019., 366(6464):447–453. ↑
Erin Smith and Helen Vogell, “How Your Shadow Credit Score Could Decide Whether You Get an Apartment” ProPublica (Mar. 29, 2022), https://www.propublica.org/article/how-your-shadow-credit-score-could-decide-whether-you-get-an-apartment. While the mere compilation and provision of background information is not subject to this bill, processing that information through an algorithm to produce a simplified output such as a score or yes/no recommendation for a prospective tenant – thereby outsourcing human discretionary decisionmaking – generally falls under this bill. ↑
“Attorney General Bonta Submits Comment Letter Recommending Reforms to the Tenant Screening Process” (May 31, 2023), https://oag.ca.gov/news/press-releases/attorney-general-bonta-submits-comment-letter-recommending-reforms-tenant#:~:text=OAKLAND. ↑
Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, “Machine Bias,” Propublica, (May 23, 2016), https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” supra, p. 420. ↑
See Mathews v. Eldridge (1976) 424 U.S. 319, 333; Daniel Keats Citron, Technological Due Process (2007) 85 Wash. U. L. Rev. 1249, 1249 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1012360 ↑
Boracas et al., Fairness and Machine Learning: Limitations and Opportunities (2023) pp. 213-214. ↑
Brian Christian, “The Alignment Problem: Machine Learning and Human Values” (Norton 2020, First Ed.), pp. 82-84. ↑
See Hilke Schellmann, “The Algorithm: How AI Decides Who Gets Hired, Monitored, Promoted, and Fired and Why We Need to Fight Back Now” (1st ed. 2024). ↑
Snake Oil, supra, at p. 73. ↑
Id. at p. 51. ↑
Id. at p. 50. ↑
Wong et al, “External valiation of a widely implemented proprietary sepsis prediction model in hospitalized patients” JAMA Int. Med. 181 (Aug. 2021), 1065–1070, https://doi.org/10.1001/jamainternmed.2021.2626. ↑
Snake Oil, supra, at p. 9. ↑
Quote from Yogi Berra. ↑
Snake Oil, supra, at p. 73. ↑
Id. at p. 80. ↑
Angelina Wang et al. 2023. Against predictive optimization: On the legitimacy of decision-making algorithms that optimize predictive accuracy. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA: ACM, 2023), 626–26. ↑
Snake Oil, 15. ↑
“RealPage Release AI Screening,” (Jun. 26, 2019), https://www.realpage.com/news/realpage-releases-ai-screening/. Emphasis in original. ↑
Snake Oil, supra, 25. ↑
Blueprint, supra, p. 14. ↑
Id. at p. 15. ↑
Ibid. ↑
Id. at p. 23. ↑
Id. at p. 27. ↑
Id. at p. 31. ↑
Id. at p. 40. ↑
Id. at p. 46. ↑
That version was, however, specific to employment, whereas prior versions applied to a broader set of contexts. ↑
Liapes v. Facebook, Inc. (2023) 95 Cal.App.5th 910, 926. ↑
See Selinger et al, “AI Audits: Who, When, How...Or Even If?” (Sep. 2023), in Collaborative Intelligence: How Humans and AI are Transforming our World, MIT Press (Forthcoming), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4568208. ↑
Blueprint, supra, p. 28; see also p. 19 (third-party auditors to demonstrate safety and effectiveness of system). ↑
E. Mulvaney, “NYC Targets Artificial Intelligence Bias in Hiring Under New Law,” Bloomberg Law, 2021, https://news.bloomberglaw.com/daily-labor-report/nyc-tar gets-artificial-intelligence-bias-in-hiring-under-new-law. ↑
Discussed in Hearing
Senate Standing Committee on Appropriations
Senate Standing Committee on Judiciary
Assembly Floor
Assembly Standing Committee on Judiciary
Assembly Standing Committee on Privacy and Consumer Protection
Bill Author
Bill Co-Author(s):