The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal, and regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.

The California Fair Employment and Housing Act protects and safeguards the right and opportunity of all persons to seek, obtain, and hold employment without discrimination, abridgment, or harassment on account of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, reproductive health decisionmaking, or veteran or military status. The act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.

This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to, on or before January 1, 2025, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 60 days of its completion and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, on and after January 1, 2026, would authorize a person certain public attorneys, including the Attorney General, to bring a civil action against a deployer or developer for a violation of the bill. The bill would require a person public attorney to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations. violations, if, among other things, the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.

This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.

This bill would prohibit a deployer from using an automated decision tool in a manner that contributes to that results in algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law. The bill would, on and after January 1, 2026, authorize a person to bring a civil action against a deployer or developer for a violation of that provision.

This bill would define deployer and developer to include a local government agency and would thereby impose a state-mandated local program.

The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal, and regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.

The California Fair Employment and Housing Act protects and safeguards the right and opportunity of all persons to seek, obtain, and hold employment without discrimination, abridgment, or harassment on account of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, reproductive health decisionmaking, or veteran or military status. The act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.

This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to, on or before January 1, 2025, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 60 days of its completion and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, on and after January 1, 2026, authorize a person to bring a civil action against a deployer or developer for a violation of the bill. The bill would require a person to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations.

This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.

This bill would prohibit a deployer from using an automated decision tool in a manner that contributes to algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.

This bill would define deployer and developer to include a local government agency and would thereby impose a state-mandated local program.

The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal, and regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.

The California Fair Employment and Housing Act protects and safeguards the right and opportunity of all persons to seek, obtain, and hold employment without discrimination, abridgment, or harassment on account of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, reproductive health decisionmaking, or veteran or military status. The act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.

This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to, on or before January 1, 2025, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 60 days of its completion and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department.

This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision. decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.

This bill would prohibit a deployer from using an automated decision tool in a manner that contributes to algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.

The California Fair Employment and Housing Act, Act protects and safeguards the right and opportunity of all persons to seek, obtain, and hold employment without discrimination, abridgment, or harassment on account of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, reproductive health decisionmaking, or veteran or military status. The act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.

Existing law establishes within the Government Operations Agency the Department of Technology, which is supervised by the Director of Technology. Existing law authorizes the director and the department to exercise various powers in creating and managing the information technology policy of the state.

Existing law, the California Fair Employment and Housing Act, protects and safeguards the right and opportunity of all persons to seek, obtain, and hold employment without discrimination, abridgment, or harassment on account of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, reproductive health decisionmaking, or veteran or military status.

This bill would state the intent of the Legislature to enact legislation relating to the use of artificial intelligence in accordance with prescribed principles relating to safety and effectiveness, algorithmic discrimination, notice and explanation of use, and human alternatives, consideration, or fallback.

Date of Hearing: May 17, 2023

ASSEMBLY COMMITTEE ON APPROPRIATIONS

Chris Holden, Chair

ABPCA Bill Id:AB 331

Author:(Bauer-Kahan) – As Amended Ver:April 19, 2023

Urgency: No State Mandated Local Program: Yes Reimbursable: Yes

SUMMARY:

This bill prohibits “algorithmic discrimination,” that is, use of an automated decision tool to contribute to unjustified differential treatment or outcomes that may have a significant effect on a person’s life.

Specifically, this bill:

1. Prohibits use of an “automated decision tool”—meaning a system or service that uses artificial intelligence (AI) and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling factor in making, “consequential decisions”—that results in algorithmic discrimination.
   1. Prohibits a person or entity that deploys an automated decision tool from using such a tool that results in algorithmic discrimination.
   2. Provides for civil action against a deployer of an automated decision tool that results in algorithmic discrimination; tasks the plaintiff with the burden to prove harm; and makes a deployer liable for compensatory damages, declaratory relief and reasonable attorney’s fees and costs.
2. Makes numerous requirements of those who develop or deploy an automated decision tool.
   1. Requires a developer of an automated decision tool to provide a deployer of the tool with a statement regarding the intended uses of the tool and documentation regarding, generally, the tool’s known limitations, including reasonably foreseeable risks of algorithmic discrimination from its intended use, date used to train or program the tool and how the tool was evaluated for validity and explainability.
   2. Requires those who develop or deploy automatic decision tools to assess them.
      1. Requires an individual or entity that deploys or develops an automated decision tool to, on or before January 1, 2025, and annually thereafter, perform and “impact assessment” of the tool that, generally, states the purpose of the tool, its uses and benefits, its outputs and their uses, and its role in making consequential decisions; summarizes the type of data collected and used by the tool in making consequential decisions; analyzes potential harms use of the tool may inflict on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status or genetic information; and describes safeguards to address any foreseeable risks of algorithmic discrimination, how the tool will be used by a “natural person” in making a consequential decision, and how the tool has been or will be evaluated for validity or relevance.
      2. Requires a developer or a deployer of an automated decision tool to perform, as soon as feasible, an impact assessment of any significant update to the automated decision tool.
      3. Exempts a deployer of an automated decision tool from the assessment requirements described in a) and b) if the developer has fewer than 25 employees, unless, as of the end of the prior calendar year, the deployer deployed an automatic decision tool that effected more than 999 people per year.
   3. Requires a program to govern use of an automated decision tool that entails, among other things:
3. A developer or deployer of an automated decision tool establishing, documenting, implementing and maintaining a governance program that includes reasonable safeguards to generally manage the reasonably foreseeable risks of algorithmic discrimination associated with use of the tool.
4. The developer or deployer designating at least one employee to be responsible for overseeing and maintaining the governance program, and the developer or deployer conducting a prompt and complete assessment of any compliance issues raised by the employee.
5. Identifying and implementing safeguards against reasonably foreseeable risks of algorithmic discrimination from use of the tool.
   1. Requires a developer or deployer of an automated decision tool to make publicly available, in a readily accessible manner, a clear policy that provides a summary of (a) the types of automated decision tools currently in use or made available to others by the deployer or developer and (b) how the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
   2. Requires a deployer of an automated decision tool used to make a consequential decision, based solely on the output of the tool, to notify a person subject to use of the tool and requires the deployer to, if technically feasible, accommodate a person’s request to not be subject to use of the automated decision tool and, instead, be subject to an alternative process.
6. Provides the California Civil Rights Department (CCRD) with enforcement action.
   1. Requires a deployer or developer to provide an impact assessment to CCRD within 60 days of its completion.
   2. Provides an administrative fine of not more than $10,000 per violation in an administrative enforcement action brought by the CCRD.
   3. Specifies that each day an automated decision tool is used for which an impact assessment has not been submitted to CCRD shall give rise to a distinct violation.
7. Authorizes the Attorney General and local prosecutorial authorities to bring civil action against a developer or deployer for any violation of the requirements of this bill and provides for the following penalties—injunctive relief, declaratory relief and reasonable attorney’s fees and litigation costs—and provides such a developer or deployer a 45-day period in which to cure any violations.

FISCAL EFFECT:

1. Significant cost to CCRD to (a) accept impact assessments submitted by developers and deployers of automated decision tools and (b) establish processes through which it assesses and collects fines. CCD estimates its costs to be in range of $17 million and $20 million annually for between 45 and 60 permanent employees. CCD warns it does not a have a reliable estimate of the number of entities that will submit impact assessments to it, so its estimates of cost are somewhat speculative.
2. Potential revenue of an unknown, but likely significant amount, from imposition of fines for violations of the bill’s provisions (General Fund).
3. Potential cost pressures (GF or Trial Court Trust Fund (TCTF)) in the tens of thousands of dollars annually to the trial courts in increased workload, given this bill creates a new crime.  A defendant charged with a misdemeanor is entitled to no-cost legal representation and a jury trial.  If five new crimes are filed annually statewide and proceed to trial resulting in two days of court time, at an estimated cost of approximately $8,000 for an eight-hour court day, the approximate annual cost to the courts is $80,000.  Although courts are not funded on the basis of workload, increased pressure on staff and the TCTF may create a need for increased court funding from the GF to perform existing duties.  The Governor’s 2023-24 state budget already proposes $109.3 million in ongoing GF revenue to continue backfilling the TCTF for expected revenue declines.
4. Potential non-reimbursable costs in the tens of thousands of dollars annually for increased incarceration costs, as misdemeanors are generally punishable by up to six months in county jail or $1,000 in fines.  The average annual cost to incarcerate a defendant in county jail is approximately $29,000.  California county jails are increasingly overcrowded – particularly since the enactment of the Realignment Act in 2011.  Jail overcrowding since 2011 has resulted in court-mandated population caps and increased rates of inmate violence and death.  Overcrowding has also resulted in significant state funding for additional jail construction.  Although new crimes are not considered reimbursable state mandates pursuant to Proposition 30 (2012), overcrowding in county jails creates cost pressure on the GF because the state has historically granted new funding to counties to offset overcrowding after the 2011 Realignment Act.

COMMENTS:

1. Purpose. The author intends this bill to protect individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools that make consequential decisions and mitigate any discovered biases. According to the author:

AB 331 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADTs has become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.

1. Background. Automated decision tools, generally, are computer-enabled tools that help people and organizations to make decisions. This bill, however, defines automated decision tools more specifically. Under this bill, an automated decision tool is a system or service that uses AI and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling factor in making, consequential decisions, with consequential decisions being those affecting an individual in regards to employment and pay, education and training, financial decisions, housing, health care, insurance, criminal justice and other subjects that may have a similarly significant effect on a person’s life.

Automated decision tools, and AI more generally, are used ever more frequently, and in ever more sophisticated ways, to help make decisions both minor and, as noted by the author, consequential. In response, the White House recently released its Blueprint for an AI Bill of Rights, which describes the use of AI and automated decision tools, as follows:

Among the great challenges posed to democracy today is the use of technology, data, and automated systems in ways that threaten the rights of the American public. Too often, these tools are used to limit our opportunities and prevent our access to critical resources or services. These problems are well documented. In America and around the world, systems supposed to help with patient care have proven unsafe, ineffective, or biased. Algorithms used in hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed new harmful bias and discrimination. Unchecked social media data collection has been used to threaten people’s opportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or consent.

The White House AI Blueprint establishes five principals to govern use of AI and automated decision tools. The Blueprint summarizes those principals as:

• You should be protected from unsafe or ineffective systems.
• You should not face discrimination by algorithms and systems should be used and designed in an equitable way.
• You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used.
• You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. 
• You should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.

The Blueprint goes on to list ways in which to apply the principals enumerated in the AI Bill of Rights.

This bill does not mirror the AI Bill of Rights; it is, however, consistent with them. The bill defines algorithmic discrimination and prohibits it. The bill requires those who develop or deploy automated decision tools to assess the tools’ potential for such discrimination. The bill requires any person subject to the use of automated decision tools in making consequential decisions about that person’s life be aware of the use and be able to opt out of it. And the bill provides legal remedy for a person who it subject to algorithmic discrimination and legal consequences for those who engage in or enable it.

1. Support and Opposition. This bill is supported by several organizations that note the widespread use of AI and automated decisions tools and who express concern over the potential—and history—of AI and these tools to incorporate unjust bias and make wrongfully discriminatory decisions. For example, the California Hawaii State Conference of the NAACP references reporting of an algorithm that “used healthcare spending as a proxy for health needs and falsely concluded that Black patients were healthier than equally sick white patients, depriving Black patients from needed high-risk care.”

The bill is opposed by a long list of organization, mostly representing various industries and business interests, that express concern the bill is too broad and will result in “overregulation in this space” that will “undermine many beneficial uses of [automated decision tools]” such as reducing “instances and effects of human bias.”

Analysis Prepared by: Jay Dickenson / APPR. / (916) 319-2081

Date of Hearing: April 18, 2023

ASSEMBLY COMMITTEE ON JUDICIARY

Brian Maienschein, Chair

ABPCA Bill Id:AB 331

Author:(Bauer-Kahan) – As Amended Ver:April 13, 2023

As Proposed to be Amended

SUBJECT: Automated decision tools

KEY ISSUES:

1. SHOULD THE CIVIL RIGHTS DEPARTMENT BE AUTHORIZED TO SEEK ADMINISTRATIVE PENALTIES OF UP TO $10,000 PER VIOLATION FOR THE FAILURE OF A DEPLOYER OR DEVELOPER, AS DEFINED, TO SUBMIT AN IMPACT ASSESSMENT?
2. SHOULD THE LEGISLATURE CREATE A PRIVATE RIGHT OF ACTION AGAINST A DEPLOYER WHO DEPLOYS AN AUTOMATED DECISION TOOL THAT RESULTS IN ALGORITHMIC DISCRIMINATION and causes the plaintiff actual harm?
3. SHOULD THE LEGISLATURE CREATE A CAUSE OF ACTION AND AUTHORIZE ENFORCEMENT BY SPECIFIED PUBLIC PROSECUTORS FOR A DEPLOYER OR DEVELOPERS VIOLATION OF ANY OF THE SPECIFIED REQUIREMENTS OF THIS BILL?

SYNOPSIS

For years, automated decision tools (ADT) (also known as automated systems, or automated decision technology) have integrated into our lives. Briefly defined, ADT are technological tools that make individualized decisions based on a coding structure. While the code and development of the automated technology is manmade, the ultimate decision is theoretically void of any human input. Most of us take automated technology and its algorithms for granted, often never recognizing when they are impacting our day to day lives. While this ensures that we can move about our lives unimpeded, it also means that we are often unaware of the prevalence of this relatively new technology. Like most any new development, automated technology has both positive and negative ramifications, and the degree of benefit or detriment that they may bring varies greatly based on the communities targeted.

This bill proposes a new framework to regulate the development and deployment of automated decision tools, seemingly modeled off of the White House’s 2022 AI Bill of Rights. Within the framework, the bill requires deployers and developers, as defined, to develop annual impact assessments, focused on the potential adverse and beneficial impacts of their ADT, and prohibits either from using an ADT that contributes to algorithmic discrimination. Within the jurisdiction of this Committee, the bill imposes various enforcement mechanisms, including an administrative enforcement mechanism for the Civil Rights Department, and a private right of action for enforcement of any of the bill’s requirements. In order to tailor the current enforcement mechanisms to more appropriately address distinct violations, the author proposes various amendments, discussed further in the comments section of the analysis.

This bill is supported by the Algorithmic Justice League, the California-Hawai’i State Conference of the NAACP, the Israeli-American Civic Action network, Oakland Privacy, and the Santa Monica Democratic Club. It is opposed by a broad coalition of business and tech industry advocates. The California Association of Realtors have submitted a position of oppose unless amended, and the California Nurses Association have submitted a position of concern. This bill was previously heard by the Assembly Committee on Privacy and Consumer Protection, and passed out on a vote of 8-3.

SUMMARY: Establishes a statutory framework for the development and use of automated decision tools (ADTs). Specifically, this bill:

1. Defines the following for purposes of this bill:
   1. “Algorithmic discrimination” means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
   2. “Artificial intelligence” means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing a real or virtual environment.
   3. “Automated decision tool” means a system or service that uses artificial intelligence and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling factor in making, consequential decisions.
   4. “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to the impact of, access to, or the cost, terms, or availability of, any of the following:
      1. Employment, workers management, or self-employment, including, but not limited to, all of the following:
         1. Pay or promotion;
         2. Hiring or termination;
         3. Automated task allocation;
      2. Education and vocational training, including, but not limited to, all of the following:
         1. Assessment, including, but not limited to, detecting student cheating or plagiarism;
         2. Accreditation;
         3. Certification;
         4. Admissions;
         5. Financial aid or scholarships;
      3. Housing or lodging, including rental or short-term housing or lodging;
      4. Essential utilities, including electricity, heat, water, internet or telecommunications access, or transportation;
      5. Family planning, including adoption services or reproductive services, as well as assessments related to child protective services;
      6. Health care or health insurance, including mental health care, dental, or vision;
      7. Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor;
      8. The criminal justice system, including but not limited to, all of the following:
         1. Risk assessments for pretrial hearings;
         2. Sentencing;
         3. Parole;
      9. Legal services, including private arbitration or mediation;
      10. Voting; and
      11. Access to benefits or services or assignments of penalties.
   5. “Deployer” means a person, partnership, state or local government agency, or corporation that uses an automated decision tool to make a consequential decision.
   6. “Developer” means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a controlling factor in making, consequential decisions, whether for its own use or for use by a third party.
   7. “Impact assessment” means a documented risk-based evaluation of an automated decision tool that meets the criteria of specified existing provisions of the Business and Professions Code.
   8. “Sex” includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
   9. “Significant update” means a new version, new release, or other update to an automated decision tool that includes changes to its use case, key functionality, or expected outcomes.
2. Requires, by January 1, 2025, and annually thereafter, a deployer of an automated decision tool to perform an impact assessment for any automated decision tool the deployer uses that includes all of the following:
   1. A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts;
   2. A description of the automated decision tool’s outputs and how they are used to make, or be a controlling factor in making, a consequential decision;
   3. A summary of the type of data collected from natural persons and processed by the automated decision tool when it is used to make, or be a controlling factor in making, a consequential decision;
   4. A statement of the extent to which the deployer’s use of the automated decision tool is consistent with or varies from the statement required by the developer in 3).
   5. An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployer’s use of the automated decision tool;
   6. A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination rising from the use of the automated decision tool known to the deployer at the time of the impact assessment;
   7. A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a controlling factor in making, a consequential decision; and
   8. A description of how the automated decision tool has been or will be evaluated for validity or relevance.
3. Requires, on or before January 1, 2025, and annually thereafter, a developer of an automated decision tool to complete and document an assessment of any automated decision tool that it designs, codes, or produces that includes all of the following:
   1. A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts;
   2. A description of the automated decision tool’s outputs and how they are used to make, or be a controlling factor in making, a consequential decision;
   3. A summary of the type of data collected from the natural persons and processed by the automated decision tool when it used to make, or be a controlling factor in making, a consequential decision;
   4. An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployer’s use of the automated decision tool;
   5. A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool; and
   6. A description of how the automated decision tool can be used by a natural person, or monitored when it used, to make, or be a controlling factor in making, a consequential decision.
4. Requires a deployer or developer, in addition to the impact assessment required by 2) and 3), to perform, as soon as feasible, an impact assessment with respect to any significant update.
5. Makes the requirements in 2) through 4) applicable only to a deployer with 25 or more employees, unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacts more than 999 people per year.
6. Requires a deployer, at or before the time an automated decision tool is used to make a consequential decision, to notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision.
   1. Requires a deployer required to provide a notice pursuant to 6) to provide all of the following:
      1. A statement of the purpose of the automated decision tool;
      2. Contact information for the deployer;
      3. A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision.
7. Requires a deployer, if a consequential decision is made solely based on the output of an automated decision and if technically feasible, to accommodate a natural person’s request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation.
   1. Following a request pursuant to 7), authorizes a deployer to reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide the information, the deployer is no longer obligated to provide an alternative selection process or accommodation.
8. Requires a developer to provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following:
   1. The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use;
   2. A description of the type of data used to program or train the automated decision tool;
   3. A description of how the automated decision tool was evaluated for validity and explainability before sale or licensing.
9. Exempts the disclosure of trade secrets from the requirement in 8) pursuant to existing law.
10. Requires a deployer or developer to establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
    1. Requires the safeguards required by 10) to be appropriate to all of the following:
       1. The use or intended use of the automated decision tool;
       2. The deployer’s or developer’s role as a deployer or developer;
       3. The size, complexity, and resources of the deployer or developer;
       4. The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool; and
       5. The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
11. Requires the governance program required by 10) to be designed to do all of the following:
    1. Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter, who shall have the authority to assert to the employee’s employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter;
    2. Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool;
    3. If established by a deployer, provide for the performance of impact assessments as required by 2).
    4. If established by a developer, provide for compliance with the requirements of 6) and 8).
    5. Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with the provisions of this bill;
    6. Maintain for two years after completion the results of an impact assessment;
    7. Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
12. Makes the requirements in 10) and 11) applicable only to a deployer with 25 or more employees, unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
13. Requires a deployer or developer to make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
    1. The types of automated decision tools currently in use or made available to others by the deployer or developer; and
    2. How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
14. Prohibits a deployer from using an automated decision tool that results in algorithmic discrimination.
15. Authorizes a person to bring a civil action, beginning January 1, 2026, against a deployer for violation of 14). Requires the plaintiff to demonstrate that the deployer’s use of the automated decision tool resulted in algorithmic discrimination that caused actual harm to the person bringing the civil action.
16. Makes a deployer liable to a prevailing plaintiff in a claim brought pursuant to 15) for compensatory damages, declaratory relief, and reasonable attorney’s fees and costs.
17. Requires a deployer or developer to provide the impact assessment to the Civil Rights Department (CRD) within 60 days of completing the impact assessment.
18. Makes a deployer or developer who fails to submit an impact assessment pursuant to 2) and 3) liable for an administrative fine of not more than $10,000 per violation in an administrative enforcement action brought by the CRD.
    1. Makes each day on which an automated decision tool is used for which an impact assessment has not been submitted a distinct violation.
19. Authorizes the CRD to share impact assessments with other state entities as appropriate.
20. Authorizes specified public attorneys to enforce the provisions of this bill through civil claims brought against a deployer or developer for violation of any requirement established by this bill.
    1. Authorizes recovery to a prevailing public attorney in a claim brought pursuant to 18) for any of the following:
       1. Injunctive relief;
       2. Declaratory relief;
       3. Reasonable attorney’s fees and litigation costs.
21. Requires a public attorney, before commencing an action pursuant to 18) for injunctive relief, to provide 45 days’ written notice to a deployer or developer of the alleged violations. If the developer or deployer cures the noticed violation and provides the prosecutor with an express written statement, made under the penalty of perjury, that the violation has been cured and that no further violations shall occur, prohibits a claim for injunctive relief from being maintained.

EXISTING LAW:

1. Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Government Code Section 12930.)
2. Enacts the Fair Employment and Housing Act. (Government Code Sections 12900 et seq.)
3. Enacts the Unruh Civil Rights Act. (Civil Code Section 51.)
4. Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
   1. Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
   2. Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civil Code Section 3426.1 (d).)

FISCAL EFFECT: As currently in print this bill is keyed fiscal.

COMMENTS: For years, automated decision tools (ADT) (also known as automated systems, or automated decision technology) have integrated into our lives. Briefly defined, ADT are technological tools that make individualized decisions based on a coding structure. While the code and development of the automated technology is manmade, the ultimate decision is theoretically void of any human input. Most of us take automated technology and its algorithms for granted, often never recognizing when they are impacting our day to day lives. While this ensures that we can move about our lives unimpeded, it also means that we are often unaware of the prevalence of this relatively new technology. Like most any new development, automated technology has both positive and negative ramifications, and the degree of benefit or detriment that they may bring varies greatly based on the communities targeted.

We are now faced with the question of how to apply existing federal and state legal protections to the world of ADT. In theory, ADT is a technology immune from the misperceptions and human prejudices. Unfortunately, due to its development by humans, ADT is often imbued with humans’ inherent biases. These biases include preconceptions based on race, gender, sex, geographic origin, and any other number of characteristics that humans themselves have developed over generations.

This bill attempts to mirror the White House’s AI Bill of Rights and ensure ADT is developed and deployed in a responsible manner. In October 2022, the White House released a white paper detailing policy proposals on how to ensure that artificial intelligence (AI), including ADT, work best for everyone. Titled The Blueprint for an AI Bill of Rights, the paper identifies five main discussion points: safe and effective systems; algorithmic discrimination protections; data privacy; notice and explanation; and human alternatives, consideration, and feedback. (The Blueprint for an AI Bill of Rights, The White House (October 2022) available at https://www.whitehouse.gov/ostp/ai-bill-of-rights/.) Recognizing the breadth of potential technology that may be captured in the discussion of the AI Bill of Rights, the white paper applies a two part test to narrow its scope to: 1) automated systems that 2) have the potential to meaningfully impact the American public’s rights, opportunities, or access to critical resources or services. (Id. at p. 8.) The paper further identifies rights, opportunities, or access to mean 1) civil rights, civil liberties, and privacy, 2) equal opportunities, and 3) access to critical resources or services. (Ibid.) The White House’s AI Bill of Rights also includes definitions for numerous phrases and concepts implicated by the paper, including algorithmic discrimination and automated system. (Id. at p.10.) The AI Bill of Rights then goes on to lay out expectations for each of the five principles, and what should be expected by the public with regard to each principle. For example, in its discussion of automated systems, the paper posits that “[t]he public should be consulted in the design, implementation, deployment, acquisition and maintenance phases of automated system development,” and that

“[b]efore deployment, and in a proactive and ongoing manner, potential risks of the automated system should be identified and mitigated. Identified risks should focus on the potential for meaningful impact on people’s rights, opportunities, or access and include those to impacted communities that may not be direct users of the automated system, risks resulting from purposeful misuse of the system, and other concerns identified via the consultation process.” (Id. at p.18.)

The paper also suggests that

“automated systems should have ongoing monitoring procedures, including recalibration procedures, in place to ensure that their performance does not fall below an acceptable level over time, based on changing real-world conditions or deployment contexts […] and should include continuous evaluation of performance metrics and harm assessments, updates of any system, and retraining of any machine learning models as necessary[.]” (Id. at p.19)

Jumping to the AI Bill of Rights’ section on “Notice and Explanation,” the paper states: “You should know that an automated system is being used, and understand how and why it contributes to outcomes that impact you.” (Id. at p. 40.)

It appears that this bill is largely modeled off of the White House’s AI Bill of Rights, as evidenced by the various reporting and notice requirements included in its language. However, it should be noted that the AI Bill of Rights appears significantly more extensive, and expands on numerous additional policy proposals – including an in-depth discussion of how to appropriately capture data relating to the disparate impact of ADT on complex issues of race or religion – that do not appear to be fully incorporated into this bill.

This bill proposes a new framework to regulate the development and deployment of automated decision tools. The bill would impose numerous requirements on both developers (those who build ADT) and deployers (those who implement ADT) of ADT. First, the bill requires deployers and developers to submit annual impact assessments to the Civil Rights Department (CRD), beginning January 1, 2025. Deployers and developers who have less than 25 employees would be exempted from this requirement, unless the ADT they have either built or used impacted more than 999 individuals in the previous calendar year. Second, in the event a consequential decision, as defined, is made only on the output of an automated decision tool, the bill requires deployers to notify the individual affected and, if possible, accommodate their request to not be subject to the ADT. Third, the bill requires deployers and developers to establish a governance program designed to reasonably foresee the risks of algorithmic discrimination associated with the use, or intended use, of the ADT in question. Fourth, deployers and developers are required to make a clear artificial intelligence policy, including a summary of both the type of ADT currently in use and how they reasonably foresee the risks of algorithmic discrimination arising as a result of the ADT it currently deploys or provides to others, and make the policy readily available to the public. Finally the bill prohibits a deployer from using an automated decision tool in a manner that contributes to algorithmic discrimination.

The bill incorporates various enforcement mechanisms to ensure the efficacy of the new reporting and policy development requirements, which are discussed further below.

The scope of this analysis. This bill was primarily referred to the Assembly Committee on Privacy and Consumer Protection. The jurisdiction of the previous committee includes the policy impacts of the bill relating to the technology of ADT and feasibility of capturing the requested information, such as which industries should be included or excluded, what form the impact assessments should take, and which agency was the appropriate one to receive the assessments.

The scope of this Committee focuses on how any rights or requirements imposed by this bill would be enforced through the courts. Namely, the private rights of action and enforcement through the CRD. As such, this analysis will not focus on anything beyond the enforcement mechanisms. While the opposition has raised concerns regarding the viability of the policy proposals included in this bill, particularly as applied to their industry, the previous committee made minimal amendments to the language and this Committee sees no need to second guess their expertise.

Clarifying the enforcement mechanisms proposed by this bill. Under the bill currently in print there are three separate enforcement mechanisms.

First, the bill establishes that a deployer or developer of an ADT may be subject to an administrative enforcement action by the CRD for failure to submit the required impact assessment. The bill authorizes CRD to collect an administrative penalty of no more than $10,000 per violation. The bill further defines each individual day that an ADT is used for which an impact assessment has been not submitted as an individual violation. This provision arguably exposes deployers and developers to significant administrative penalties for failure to submit an impact assessment. Considering the importance of the impact assessments in understanding the effect of ADT, these high penalties are arguably justifiable. However, considering the bulk of the information required to be submitted by deployers and developers regarding the benefit or risks surrounding ADTs will be housed with CRD, thus making it the agency with the most intimate knowledge of potential violations, the author may wish to consider measures to increase CRD’s enforcement capabilities.

Second, as currently written, this bill prohibits a deployer from using an ADT “in a manner that contributes to algorithmic discrimination.” It is not clear, however, how a plaintiff would establish the “manner” in which the ADT was used, or how a defendant would demonstrate that an ADT did not “contribute” to algorithmic discrimination. While the ultimate intent may be clear to someone discussing the impacts of ADT – that is to avoid widespread use of tools that are imbued with bigoted proclivities –this language is arguably impractical. In order to address this concern, the author proposes to amend the language to prohibit a deployer from using an automated decision tool that results in algorithmic discrimination. Further, the author proposes requiring a plaintiff bringing a civil action under this language to bear the burden of demonstrating that they suffered actual harm as a result of the ADT’s algorithmic discrimination. The intent of this language is to ensure that an individual who utilizes the ADT simply to demonstrate its tendency to engage in algorithmic discrimination, with no actual intent to use the ADT for its intended purpose, is not authorized to bring a claim. In combination, these amendments ensure that the root of the issue – the deployment of technology that results in discriminatory outcomes – is captured. Additionally, the author proposes delaying implementation of this private right of action until January 1, 2026, to allow businesses the time necessary to identify ADT which do result in discriminatory outcomes and cease their use. Finally, the author proposes clarifying the remedies which may be recovered by a violation of this section. The amendments would be as follows:

22756.6.   (a)  A deployer shall not use an automated decision toolin a manner that contributes to results in algorithmic discrimination.

(b) (1) On and after January 1, 2026, a person may bring a civil action against a deployer for violation of this section.

(2) In an action brought pursuant to paragraph (1), the plaintiff shall have the burden of proof to demonstrate that the deployer’s use of the automated decision tool resulted in algorithmic discrimination that caused actual harm to the person bringing the civil action.

(c) In addition to any other remedy at law, a deployer that violates this section shall be liable to a prevailing plaintiff for any of the following:

(1) Compensatory damages.

(2) Declaratory relief.

(3) Reasonable attorney’s fees and costs.

Third, the bill as currently written creates a private right of action for any individual to bring a civil claim against a deployer or developer for a violation of any provision of this bill. This is arguably unduly expansive, and impractical. While it is reasonable to grant an individual a private right of action for use of an ADT which results in algorithmic discrimination, as that application is likely to result in harm to the plaintiff themselves, it is less reasonable to grant an individual the right to sue a business for their failure to submit an impact assessment to CRD, or any of the other numerous requirements imposed by this bill. For one thing, it is not clear how an individual would be aware that a deployer or developer failed to submit their impact assessment. For another, it is arguably an imbalanced approach to allow for every individual to bring a claim against an individual deployer or developer for their failure to submit an impact assessment when there is no guarantee that any of those individuals would ever come into contact with the deployer or developer. What appears more reasonable is to authorize public prosecutors to bring a civil claim for a deployer or developer’s violation of any of the provisions of this bill. With this amendment, the bill appears more appropriately tailored, and authorizes the state to seek enforcement, and production of, data that is ultimately due to them. Finally, the author proposes several clarifying amendments to the paragraph requiring notice to the developer or deployer. The proposed amendments are as follows:

22756.8.   (a) (1) On and after January 1, 2026, a person Any of the following public attorneys may bring a civil action against a deployer or developer for a violation of this chapter. chapter:

(A) The Attorney General in the name of the people of the State of California.

(B) A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.

(C) A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.

(2) A court may award to a prevailing plaintiff in an action brought pursuant to this subdivision all of the following:

(A) Compensatory damages.

(A) Injunctive relief.

(B) Declaratory relief.

(C) Reasonable attorney’s fees and litigation costs.

(b) (1) (A) Subject to paragraph (2), a person, A public attorney, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days’ written notice to a deployer or developer of the alleged violations of this chapter.

(2) (A) If the The developer or deployer demonstrates to the court within may cure, within 45 days of receiving the written notice described in paragraph (1) that it has cured a noticed (1), the noticed violation and provides provide the person who gave the notice an express written statement statement, made under penalty of perjury, that the violation has been cured and that no further violations shall occur, a claim for injunctive relief shall not be maintained for the noticed violation. occur.

(B) If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.

Several opponents of the bill have voiced concern with the expansive nature of the private right of action currently written. To the extent this amendment significantly narrows the ability for private individuals to bring civil actions against deployers or developers, it is possible that this concern is at least partially addressed.

The private right of action, while potentially creating overlap with existing legal protections, nonetheless ensures full coverage of potential violations. Both the Unruh Civil Rights Act and the Fair Employment and Housing Act (FEHA) prohibit discrimination on the basis of various characteristics. The Unruh Civil Rights Act, codified in Section 51 of the Civil Code, declares that:

“all persons within the jurisdiction of the state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.”

FEHA, beginning with Section 12940 of the Government Code, provides protections in both the employment and housing contexts for an entity to discriminate “because of the race, religious creed, color, national origin, ancestry, physical disability, mental disability, reproductive health decisionmaking, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, age, sexual orientation, or veteran or military status of any person.” The private right of action in this bill creates a right of action that is arguably already provided for in both of these statutes. For example: imagine a Latino applicant named Juan Soto emails in his resume for a job directly to the employer soliciting applications. An hour later, a white male applicant named John Smith submits his resume for the same job to the same email address. Both resumes are practically identical and both applicants have completed the instructions provided for the application. The only substantive difference between the two is the name at the top – one which is apparently Latino and the other which is not. If the employer were to reject Juan Soto and offer the position to John Smith on the basis of their resumes alone, Juan Soto may very well have a cause of action under FEHA for discrimination on the basis of race, national origin, or ancestry.

The private right of action established by this bill arguably addresses this same scenario, but in a slightly different context. Say in this case, both Juan Soto and John Smith, rather than emailing in their resumes directly to the employer instead submit their resumes via an online application portal that purports to complete an initial screening of all applications using ADT. If the online application portal rather than the human employer eliminates Juan Soto’s application in its initial screening, Juan Soto may have a cause of action, so long as he is able to demonstrate use of the ADT 1) resulted in algorithmic discrimination and 2) resulted in harm. It is arguable that the rejection of his resume, identical to John Smith’s in every substantive way except his name, is evidence that use of the ADT resulted in discrimination, while the loss of a job opportunity is likely clear demonstrable harm. Therefore, the private right of action established by this bill is an important element to help close a loophole that would allow deployers to pin all blame on ADT, rather than accept any accountability, for discriminatory outcomes.

Final proposed amendments. As currently written, this bill prohibits a city or county from adopting any regulation or rule relating to ADT or an impact assessment similar to that required to be submitted to the state under this bill. This seems unnecessarily restrictive. In fact, the Legislature should arguably be encouraging local cities and counties to investigate the issue on a more individualized level. It is likely that cities and counties, who have more intimate knowledge of the businesses in their jurisdictions, are well positioned to help develop a robust statewide ADT policy. Additionally, recent amendments have expanded the bill’s reporting requirements to state or local government agency. To the extent this language was intended to ensure that state and local agencies were captured by the bill’s requirements, the recently added language has addressed the concern and thus this provision is arguably unnecessary. The author proposes to strike Section 22756.9 in its entirety.

The California Association of Realtors have submitted a position of oppose unless amended. In furtherance of their position the Realtors state: “Many small housing providers require the use of third-party software tools to help facilitate the rental application process. AB 331 could impose onerous requirements upon them, which could impede and increase the costs of the rental application process and, in turn, erode the ability of such providers to house individuals and families in a timely manner.” However, this committee’s jurisdiction is as to the enforcement mechanism, rather than the scope of who should or should not be required to abide by the requirements imposed by this bill. Therefore, this is not an issue appropriately addressed in this committee.

ARGUMENTS IN SUPPORT: This bill is supported by The California-Hawaii Conference of the NAACP, the Israeli-American Civic Action Network, and The Santa Monica Democratic Club. In support of the measure, the California-Hawaii Conference of the NAACP writes:

ADTs have been found to exhibit biases and consequently have resulted in discriminatory impacts and harm to marginalized communities. A study published in Science showed that a clinical algorithm used across hospitals for determining patient care was racially biased against Black patients. The algorithm used healthcare spending as a proxy for health needs and falsely concluded that Black patients were healthier than equally sick white patients, depriving Black patients from needed high-risk care. These results were biased because Black patients face disproportionate poverty levels and spend less on health care.

The CA/HI NAACP’s principal objective is to ensure the political, educational, social, and economic equality of minority citizens in California and eliminate race prejudice. ADTs are prominent in almost every facet of an individual’s life, and they must be used safely to avoid discriminatory harm. The mentioned restrictions provide a sense of accountability that makes it difficult for those who use ADTs to use them as tools for discrimination and oppression, whether in employment practices, housing, the criminal justice system, or any other facet of life. For these reasons, CA/HI NAACP proudly supports AB 331 (Bauer-Kahan).

Further, the Israeli-American Civic Action Network writes:

Our organization believes that technology should be used to empower people and communities, and we are committed to ensuring that technology is designed and used in a way that promotes fairness, equity, and justice. As an organization that works to empower immigrants to the United States from Israel by helping them get more engaged in advocacy and civic life, we understand the importance of protecting marginalized communities from the negative impacts of algorithmic bias and discrimination. We believe that AB 331 will help to address these issues and promote a more just and equitable society for all Californians.

Automated decision tools use statistical analyses to assess one’s eligibility for a benefit or penalty. These systems have been traditionally used for credit decisions, however usage has expanded to employment screening, insurance eligibility, and health care decisions. ADTs are also used in the public sector, including government services, and in criminal justice sentencing and probation decisions.

ADTs may also take into account factors such as neighborhood demographics. In some cases, this could lead to Jewish immigrants being unfairly denied loans if they live in neighborhoods that are unfairly characterized as higher-risk due to antisemitic stereotypes. This could ultimately lead to immigrants being unfairly denied access to credit and opportunities for financial growth. By supporting AB 331, we can work to prevent such biased outcomes and ensure that all individuals, including Jewish people, are treated fairly and equitably by automated decision tools.

ADTs are prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help

ARGUMENTS IN OPPOSITION: This bill is opposed by a coalition of business advocates and tech industry organizations. In support of their position, the California Apartment Association writes:

On behalf of the members of the California Apartment Association (CAA), I am writing to inform you that CAA has taken an oppose position on AB 331, your bill that would, among other things, require a rental property owner or management company (referred to as a deployer) to annually perform an “impact assessment” for any “automated decision tool” as defined, that the deployer uses and to provide that assessment to the Civil Rights Department. This bill would require that the deployer provide any natural person – in this case a tenant-applicant - who is the subject of an automated decision tool with, among other things, a statement of the purpose of the automated decision tool and allow the tenant-applicant to request that they not be subject to the automated decision tool.

Rental property owners and property management companies use automated tools, such as credit and eviction reports, to make decisions about a tenant-applicant’s ability to pay the rent. These “decision tools” are not used to illegally discriminate and instead are necessary business tools. Giving a tenant-applicant the ability to opt out of such a decision-making tool is not logical. The owner would have no objective way to make a decision about the applicant nor the bandwidth to institute an alternative process in the event of an opt out. We ask that the rental housing industry be excluded from the provisions of AB 331.

Further, the Chamber of Commerce writes:

Significant concerns around AB 331’s private right of action, administrative penalties, and limited right to cure.

[…]

As currently drafted, it is possible that a violation could constitute not only a failure to complete an impact assessment altogether, but could also be interpreted as any single deficiency within that impact assessment. Even then, it is unknown whether the number of violations is based on that single error, or by any single error multiplied by the number of individuals who were potentially impacted by the use of a particular type of ADT for which an assessment was required or who received an inadequate notice.

Take for example AB 331’s requirement that an impact assessment include an analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information. (See proposed section 22756.1(a)(5).) If an analysis is completed, but a potential adverse impact is left out, but not with any intent or malice, is that a violation subject to a private right of action? By way of another example, the bill requires a deployer to notify a natural person, at or before the time an ADT is used to make a consequential decision, that an ADT is being used to make or be a controlling factor in making the consequential decision. (See proposed section 22756.2.) What happens in the case of a hospital emergency room where prior notice simply is not possible? It is unclear to us what constitutes proper notice to avoid individual lawsuits alleging non-compliance. Ultimately, we are concerned that the potential for a private right of action as well as the steep fines will have a significant chilling effect on innovation in California and on access to important technology – including technologies that could reduce the instances of human bias and discrimination.

And while we support the opportunity to cure, conceptually, as drafted AB 331’s right to cure is inadequate and illusory. First, the bill authorizes a court to award to any prevailing plaintiff all of the following: compensatory damages, injunctive relief, declaratory relief, and reasonable attorney’s fees and litigation costs. The right to cure only applies to injunctive relief and it does not apply at all in the context of an administrative action. Even still, no company can realistically avail themselves of this right under AB 331, as it requires them to not only cure the noticed violation, but also provide the person giving the notice an express written statement that the violation has been cured and that no further violations shall occur. Due to the complexity and evolving nature of this technology, as well as the bill’s breadth and vagueness issues, it is unrealistic for company to be asked to sign a statement that they will never again violate this law.

Other active legislation and regulatory efforts on related topics opens the door for vast confusion and laws and regulations in conflict.

AB 331 is not the only measure related to automated decisions, artificial intelligence (AI) and issues around bias or discrimination when using such technology. To date, including AB 331, there are at least five bills and two regulatory bodies grappling with the issue in one form or another.

Of note, Proposition 24 of 2020, required the California Privacy Protection Agency to issue regulations “governing access and opt-out rights related to businesses’ use of automated decision-making technology, including profiling and requiring businesses’ response to access requirements to include meaningful information about the logic involved in these decisionmaking processes, as well as a description of the likely outcome of the process with respect to the consumer.” (Civ. Code Sec. 1798.185.) As such, just last week, the Agency ended an informal, preliminary comment period on several topics, including automated decision-making, though it has yet to commence formal rulemaking on the topic or share any draft regulations with the public.

In addition to all of this, the Civil Rights Council within the Civil Rights Department (formerly, the Department of Fair Employment and Housing) is also working on regulating the use of AI and machine learning in connection with employment decision-making. The Council recently published draft modifications to their proposed employment regulations regarding “automated decision systems” in their effort to incorporate such technology into existing rules regulating California employment and hiring practices.

With all these moving parts, it is difficult to foresee how such laws and regulations will layer on top of one another and whether there will be conflicting public policy around the use of such tools and technologies. While some approaches suggest further study and understanding, others have proceeded toward regulating the technology, at times in overlapping contexts such as employment. Understandably, our members are alarmed by the likelihood of conflict and confusion at the conclusion of these efforts that are being run in parallel to each other, without any coordination or consideration of the other efforts underway.

REGISTERED SUPPORT / OPPOSITION:

Support

Algorithmic Justice League
California-Hawai’i State Conference of the NAACP
Israeli-American Civic Action Network
Oakland Privacy
The Santa Monica Democratic Club

Opposition

American Financial Services Association
California Apartment Association
California Bankers Association
California Chamber of Commerce
California Credit Union League
California Financial Services Association
California Grocers Association
California League of Food Producers
California Manufacturers & Technology Association
California Retailers Association
Card Coalition
Civil Justice Association of California
Computer & Communications Industry Association
Insights Association
National Payroll Reporting Consortium
Netchoice
Software & Information Industry Association
State Privacy and Security Coalition, INC.
Technet

Oppose Unless Amended

California Association of Realtors

Concern

California Nurses Association

Analysis Prepared by: Manuela Boucher-de la Cadena / JUD. / (916) 319-2334

Date of Hearing: April 11, 2023

ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION

Jesse Gabriel, Chair

ABPCA Bill Id:AB 331 (

Author:Bauer-Kahan) – As Amended Ver:March 30, 2023

As Proposed to be Amended

SUBJECT: Automated decision tools

SYNOPSIS

This author-sponsored measure is a significant, meaningful attempt to set state policy with respect to private and public use of automated decision tools (ADTs): systems and services that employ artificial intelligence, and make, or are a controlling factor in making, consequential decisions. Consequential decisions, in turn, are those that materially impact the necessities of a person’s life, such as housing, education, employment, and health care (including reproductive health care), as well as the person’s involvement with critical public functions such as voting and the criminal justice system.

The bill would establish requirements for informed, conscious deployment of ADTs, based on the White House’s recently-released Blueprint for an AI Bill of Rights. Key aspects of the bill include the following:

• Developers and deployers must prepare impact statements that address intended uses and potential effects of ADTs, as well as evaluation of their validity.
• Developers and deployers must also prepare governance programs to manage the risk of algorithmic discrimination from use of ADTs.
• Deployers must notify any natural person who will be the subject of a consequential decision if an ADT will be used to make, or will be a controlling factor in making, the decision.
• A deployer must accommodate, where technically feasible, a natural person’s request to not be subject to an ADT and to instead be subject to an alternative selection process or accommodation, if a consequential decision will be made solely based on the ADT’s output.

Broadly speaking, the bill only prescribes one policy outcome: it forbids algorithmic discrimination against individuals on the basis of characteristics protected under state civil rights law, such as race, age, national origin, disability, limited English proficiency, and sex (including pregnancy, childbirth, and related conditions; gender identity; intersex status; and sexual orientation). Much of the bill is instead devoted to ensuring that ADT developers and deployers thoroughly consider the design, testing, and functioning of ADTs before releasing them into the world. This appears to be sound policy, as developers and deployers are the entities that possess the most knowledge about the systems they work with.

This bill is supported by the California-Hawaii State Conference of the NAACP, the Israeli-American Civic Action Network, Oakland Privacy, and the Santa Monica Democratic Club. It is opposed by a coalition of nineteen business trade associations, led by the California Chamber of Commerce.

If passed by this Committee, this bill will next be heard by the Assembly Judiciary Committee.

SUMMARY: Establishes a statutory framework to further the safe and informed use of automated decision tools in California. Specifically, this bill:

1. Defines the following terms:
   1. “Automated decision tool” (ADT) means a system or service that (i) uses artificial intelligence and (ii) has been specifically developed and marketed to, or specifically modified to, (iii) make, or be a controlling factor in making, consequential decisions.
   2. “Artificial intelligence” means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing a real or virtual environment.
   3. “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to the impact of, access to, or the cost, terms, or availability of, any of the following:
      1. Employment, worker management, or self-employment, including, but not limited to, pay or promotion; hiring or termination; and automated task allocation.
      2. Education and vocational training, including, but not limited to, assessment (including, but not limited to, detecting student cheating or plagiarism); accreditation; certification; admissions; and financial aid or scholarships.
      3. Housing or lodging, including rental or short-term housing or lodging.
      4. Essential utilities, including electricity, heat, water, internet or telecommunications access, or transportation.
      5. Family planning, including adoption services or reproductive services, as well as assessments related to child protective services.
      6. Health care or health insurance, including mental health care, dental, or vision.
      7. Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
      8. The criminal justice system, including, but not limited to, risk assessments for pretrial hearings; sentencing; and parole.
      9. Legal services, including private arbitration or mediation.
      10. Voting.
      11. Access to benefits or services or assignment of penalties.
   4. “Algorithmic discrimination” means the condition in which an ADT contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex (including pregnancy, childbirth, and related conditions; gender identity; intersex status; and sexual orientation), religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
   5. “Developer” means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an ADT, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a controlling factor in making, consequential decisions, whether for the developer’s own use or for use by a third party.
   6. “Deployer” means a person, partnership, state or local government agency, or corporation that uses an ADT to make a consequential decision.
   7. “Significant update” means a new version, new release, or other update to an ADT that includes changes to its use case, key functionality, or expected outcomes.
   8. “Impact assessment” means a documented risk-based evaluation of an ADT that meets the criteria set forth under 2) and 3) below.
2. Requires, beginning on or before January 1, 2025, and annually thereafter, a developer to perform an impact assessment for any ADT the developer designs, codes, or produces. The impact assessment must include all of the following:
   1. A statement of the ADT’s purpose and its intended benefits, uses, and deployment contexts.
   2. A description of the ADT’s outputs and how they are used to either make, or be a controlling factor in making, a consequential decision.
   3. A summary of the type of data collected from natural persons and processed by the ADT when it is used to make, or be a controlling factor in making, a consequential decision.
   4. An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from use of the ADT.
   5. A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the ADT.
   6. A description of how the ADT can be used by a natural person, or monitored when it is used, to make, or be a controlling factor in making, a consequential decision.
3. Requires, beginning on or before January 1, 2025, and annually thereafter, a deployer to perform an impact assessment for any ADT the deployer uses. The impact assessment must include all of the following:
   1. A statement of the ADT’s purpose and its intended benefits, uses, and deployment contexts.
   2. A description of the ADT’s outputs and how they are used to make, or be a controlling factor in making, a consequential decision.
   3. A summary of the type of data collected from natural persons and processed by the ADT when it is used to make, or be a controlling factor in making, a consequential decision.
   4. A statement of the extent to which the deployer’s use of the ADT is consistent with or varies from the statement required of the developer under 9) below.
   5. An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployer’s use of the ADT.
   6. A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination known to the deployer at the time of the impact assessment arising from the use of the ADT.
   7. A description of how the ADT will be used by a natural person, or monitored when it is used, to make, or be a controlling factor in making, a consequential decision.
   8. A description of how the ADT has been or will be evaluated for validity or relevance.
4. Requires developers and deployers to, in addition to the requirements of 2) and 3), perform as soon as possible an impact assessment with respect to any significant update, as defined.
5. Requires a deployer to notify any natural person that is the subject of a consequential decision that an ADT is being used to make, or be a controlling factor in making, the consequential decision. The notification must be made at or before the time the ADT is used.
6. Specifies that notification under 5) must include:
   1. A statement of the ADT’s purpose.
   2. The deployer’s contact information.
   3. A plain language description of the ADT, including a description of any human components and how any automated component is used to inform a consequential decision.
7. Requires the deployer, in cases in which a consequential decision is made solely based on the output of an ADT, to accommodate a natural person’s request to not be subject to the ADT and to instead be subject to an alternative selection process or accommodation—but only if this alternative is technically feasible.
8. Permits a deployer, after a request under 7) is made, to reasonably request, collect, and process information from a natural person in order to identify the person and the associated consequential decision. Absent this information, the deployer is not obligated to provide an alternative selection process or accommodation.
9. Requires a developer to provide a deployer with a statement regarding the intended uses of an ADT, as well as documentation regarding the following:
   1. The known limitations of the ADT, including any reasonably foreseeable risks of algorithmic discrimination arising from the ADT’s intended use.
   2. A description of the type of data used to program or train the ADT.
   3. A description of how the ADT was evaluated for validity and explainability before its sale or licensing.
10. Clarifies that 9) does not require the disclosure of trade secrets, as defined under the Civil Code.
11. Requires deployers and developers to establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an ADT.
12. Explains that the governance program required under 11) must be designed to do all of the following:
    1. Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this bill. This employee is authorized to assert to their employer a good faith belief that the design, production, or use of an ADT fails to comply with this bill’s requirements; if this assertion is made, the employer must conduct a prompt and complete assessment of any compliance issues that the employee raised.
    2. Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an ADT.
    3. If established by a deployer, provide for the performance of impact assessments under 3).
    4. If established by a developer, provide for compliance with 2) and 9).
    5. Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this bill.
    6. Maintain the results of impact assessments for two years after completion.
    7. Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the ADT, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
13. Exempts from the requirements of 2) and 12) a deployer with fewer than 25 employees, unless, as of the previous calendar year, the deployer deployed an ADT that impacted more than 999 people per year.
14. Requires developers and deployers to make publicly available, in a readily accessible manner, a clear policy that summarizes both of the following:
    1. The types of ADTs that the developer or deployer currently uses or makes available to others.
    2. How the developer or deployer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the ADTs that it currently uses or makes available to others.
15. Prohibits a deployer from using an ADT that contributes to algorithmic discrimination.
16. Requires a deployer or developer that is required by this bill to complete an impact assessment to provide the assessment to the Civil Rights Department within 60 days after it is completed.
17. Specifies that the administrative fine for non-compliance with the requirement under 16) is $10,000 per violation. Further specifies that each day that an ADT is used is a distinct violation if an impact assessment has not yet been submitted.
18. Permits the Civil Rights Department to share impact assessments with other state entities as appropriate.
19. Provides, beginning on January 1, 2026, a private right of action to a person against a deployer or developer for a violation of a provision of this bill.
20. Provides the following remedies to a prevailing plaintiff: compensatory damages, injunctive relief, declaratory relief, reasonable attorney’s fees, and litigation costs.
21. Requires a prospective plaintiff to provide 45 days’ written notice to a deployer or developer of an alleged violation before commencing an action for injunctive relief, except that this requirement shall not apply if the violation is substantially the same as one previously the subject of a notice.
22. Provides that if a developer or deployer receives a notice of prospective injunctive relief under 21), the developer or deployer has 45 days to provide the prospective plaintiff with an express written statement, made under penalty of perjury, that the violation has been cured and that no further violations shall occur. If the developer or deployer does so, the prospective plaintiff will no longer have a claim for injunctive relief.
23. Preempts any local law, regulation, rule, requirement, or standard related to performing an impact assessment, governance program, or other equivalent of an ADT.

EXISTING LAW:

1. Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
2. Enacts the Fair Employment and Housing Act. (Gov. Code §§ 12900 et seq.)
3. Enacts the Unruh Civil Rights Act. (Civ. Code § 51.)
4. Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
   1. Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
   2. Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civ. Code § 3426.1(d).)

FISCAL EFFECT: As currently in print this bill is keyed fiscal.

COMMENTS:

1) Background. The term “automated decision tool,” or “ADT,” refers to an ever-growing array of computer systems that are being relied upon to make, or to be a controlling factor in making, consequential decisions. ADTs are used to decide whom to hire, what treatments patients should receive, whether to grant parole, and many other life-altering decisions that used to be made entirely by human beings.

Using an ADT is, in many cases, a good thing. After all, computers can generally draw upon far more digitally-encoded data, and process this data far more quickly, than any human being. This can yield astonishing insights. A recent New York Times article described the development of a breast cancer screening system that uses artificial intelligence:

From the millions of cases the system is fed, the technology creates a mathematical representation of normal mammograms and those with cancers. With the ability to look at each image in a more granular way than the human eye, it then compares that baseline to find abnormalities in each mammogram.

Last year, after a test on more than 275,000 breast cancer cases, Kheiron [Medical Technologies, the system’s developer] reported that its A.I. software matched the performance of human radiologists when acting as the second reader of mammography scans. It also cut down on radiologists’ workloads by at least 30 percent because it reduced the number of X-rays they needed to read. In other results from a Hungarian clinic last year, the technology increased the cancer detection rate by 13 percent because more malignancies were identified. (Santariano & Metz, Using A.I. to Detect Breast Cancer That Doctors Miss (Mar. 5, 2023) N.Y. Times, available at https://www.nytimes.com/2023/03/05/technology/artificial-intelligence-breast-cancer-detection.html.)

But we must be careful to avoid what New York University Professor Meredith Broussard describes as “the notion that computers are more ‘objective’ or ‘unbiased’ because they distill questions and answers down to mathematical evaluation…an unwavering faith that if the world used more computers, and used them properly, social problems would disappear and we’d create a digitally enabled utopia.” (Broussard, Artificial Unintelligence (MIT Press 2019) p. 8.) It remains the case that many aspects of the real world are not reducible to data, and that there are many complex situations presenting competing demands which human beings remain best equipped to assess.

Recent situations when ADTs substituted for human decision-making, to the detriment of human beings, include:

• A privately-built, error-prone software system operated with minimal employee oversight by the State of Michigan wrongly accused 40,000 people of unemployment fraud between 2013 and 2015, many of whom were forced to pay heavy fines, declare bankruptcy, or have their homes foreclosed upon. Upon appeal, less than 8% of those fraud charges were validated. (de la Garza, States’ Automated Systems Are Trapping Citizens in Bureaucratic Nightmares with Their Lives on the Line (May 20, 2020) Time Magazine, available at https://time.com/5840609/algorithm-unemployment/.)
• A 54-year-old Black man was arrested in Maryland and spent nine days in jail after being misidentified by facial recognition software. (Johnson, Face Recognition Software Led to his Arrest. It Was Dead Wrong. (Feb 28, 2023) WIRED, available at https://www.wired.com/story/face-recognition-software-led-to-his-arrest-it-was-dead-wrong/.)
• A self-driving Uber car that killed a 49-year-old jaywalker in Arizona was programmed not to classify objects as pedestrians unless those objects were detected near crosswalks. (McCausland, Self-driving Uber car that hit and killed woman did not recognize that pedestrians jaywalk (Nov. 9, 2019) NBC News, available at https://www.nbcnews.com/tech/tech-news/self-driving-uber-car-hit-killed-woman-did-not-recognize-n1079281.)

As Broussard writes, “Because social decisions are about more than just calculations, problems will always ensue if we use data alone to make decisions that involve social and value judgments.” (Artificial Unintelligence, supra, p. 119.) Human judgment can mean the difference between being housed or unhoused, being free or imprisoned, and in extreme cases, between life and death.

ADT usage continues to increase. Global consulting firm McKinsey & Company reports that the adoption of artificial intelligence among the organizations it surveyed more than doubled between 2017 and 2022, with 50 percent now reporting they use the technology. (McKinsey & Co.. The state of AI in 2022—and a half decade in review (Dec. 6, 2022), available at https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022-and-a-half-decade-in-review.)

The premises of this bill are, first, that Californians ought to be much more informed about the ADTs that are being used to make consequential decisions that impact their lives, and second, that those who develop and deploy ADTs ought to be much more conscious and deliberate about what those impacts might be. In short, that we ought to get a handle on ADTs while we still can, before they become yet another phenomenon—like greenhouse gases and guns and urban sprawl—that we could have dealt with deliberately and intelligently in the past, but instead allowed to expand without any constraints, and now find ourselves struggling to contain.

2) Author’s statement.

AB 331 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADTs has become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.

3) Key terms in this bill. There are a few key terms that are first necessary to understand in order to understand what this bill would do.

The first term, of course, is “automated decision tool” (ADT), defined as a system or service that (i) uses artificial intelligence and (ii) has been specifically developed and marketed to, or specifically modified to, (iii) make, or be a controlling factor in making, consequential decisions.

The second term is “consequential decision,” which has a lengthy definition in the bill, as set forth in 1) c) in the SUMMARY above. Briefly, a consequential decision can be understood as a decision that materially impacts the necessities of a person’s life, such as education, housing, employment, or health care (including reproductive health care), as well as the person’s involvement with critical public functions, such as voting and the criminal justice system.

It is important to distinguish ADTs, as defined under this bill, from the larger category of automated decision tools that have replaced jobs that humans used to do, such as elevator or telephone switchboard operator. These tools use artificial intelligence and have been specifically developed and marketed to make decisions—but not consequential decisions.

An ADT can be a system or service that uses artificial intelligence and has been specifically developed and marketed to make consequential decisions. For example, an automated system that schedules fast food workers’ shifts and directs them as to when to appear and leave would likely qualify as an ADT under this bill, since it makes decisions as to automated task allocation.

An ADT can also be a system or service that uses artificial intelligence and has been specifically modified to make consequential decisions. The author provides the example of an ADT that is meant to evaluate whether a person convicted of a crime is likely to reoffend then being re-deployed to determine whether the person should receive access to reentry services.

An ADT can also be a system or service that uses artificial intelligence and has been specifically developed and marketed to be a controlling factor in making consequential decisions. The term “controlling factor” is not defined in the bill, which gives it the necessary flexibility to encompass a wide variety of situations. But the terms suggests that, for an ADT to be a controlling factor in a consequential decision, the output provided by the ADT must ultimately determine the decision; if the ADT had provided output that was significantly different, the consequential decision would have also turned out differently. E.g., if a loan applicant’s credit score is a factor that could outweigh all other factors (age, marital status, income) in determining whether to give the applicant a home loan, then the system that produces the credit score would be an ADT for purposes of this bill.

Two other terms bear mention: “developer” and “deployer.” As the terms suggest, a developer designs, codes, or produces ADTs, while a deployer uses ADTs. Developers and deployers may be individuals, private entities (including corporations), or state and local agencies.

4) What this bill would do. The provisions of this bill are based upon the principles outlined in the Blueprint for an AI Bill of Rights, issued by the White House Office of Science and Technology Policy on October 4, 2022. The Blueprint is available to read at https://www.whitehouse.gov/ostp/ai-bill-of-rights/.

The Blueprint identifies five principles that, as it explains, should “guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence.” These principles are Safe and Effective Systems; Algorithmic Discrimination Protections; Data Privacy; Notice and Explanation; and Human Alternatives, Consideration, and Fallback. Each of these principles, other than Data Privacy (which has been the subject of extensive prior legislation), is implemented in this measure.

• Principle: Safe and Effective Systems

The Blueprint summarizes this principle as meaning that “you should be protected from unsafe or ineffective systems.”

The bill would implement this principle by requiring developers and deployers to complete impact assessments for any ADT that they develop or use, respectively. Specific criteria for these impact assessments are set forth in the SUMMARY above, in paragraphs 2) (for developers) and 3) (for deployers). As can be seen, the impact assessments require thoughtful consideration of what ADTs are intended to do, what their effects might be, and how they are validated. Once completed, these impact assessments must be shared with the Civil Rights Department, which can then share them with other state agencies.

Both types of impact assessments would have to be completed by January 1, 2025, and annually thereafter. Bill opponents argue that “the annual nature of these assessments will be burdensome, particularly for smaller businesses.” However, the author has anticipated this objection, and exempted deployers with fewer than 25 employees from this requirement, unless such a deployer previously used an ADT that impacted more than 999 people per year.

Developers and deployers would have to complete new impact assessments any time a significant update, as defined, is made to an ADT.

Bill opponents argue that, in preparing impact statements, deployers may lack the “knowledge and expertise of the underlying technology created by [a developer] to understand all the varying uses aside from their own intended use.” The bill already requires developers to share documentation about many aspects of the ADT with deployers that should help the latter prepare impact statements. The author may wish to consider whether developers should share additional documentation beyond that already specified in the bill.

• Principle: Algorithmic Discrimination Protections

The Blueprint summarizes this principle as meaning that “you should not face discrimination by algorithms[,] and systems should be used and designed in an equitable way.”

The bill would implement this principle in at least three ways. First, developers and deployers would be required to prepare governance programs containing “reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use” of an ADT. Specific criteria for this governance program are set forth in the SUMMARY above, in paragraph 12).

Second, at least one employee would have to be designated to oversee and maintain the governance program and compliance with this bill’s requirements. That employee would have “the authority to assert to [their] employer a good faith belief that the design, production, or use of an automated decision tool fails to comply” with one or more requirements of the bill; the employer would then be required to conduct a prompt and complete assessment of the compliance issues raised.

Third, the bill would flatly prohibit a deployer from using an ADT that contributes to algorithmic discrimination on the basis of any category protected under California’s civil rights laws.

• Principle: Notice and Explanation

The Blueprint summarizes this principle as meaning that “you should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.”

The bill would implement this principle by requiring a deployer to notify any natural person that is the subject of a consequential decision that an ADT will be used to make, or will be a controlling factor in making, that decision. Requirements for the content of the notification are set forth under paragraph 6) in the SUMMARY above. The notification must be made at or before the time the ADT is used.

Bill opponents object that this notice requirement cannot be put into practice in some situations, such as if an ADT is being used to guide decision-making in an emergency room. The author may wish to clarify this requirement to address emergency situations where it is impossible to provide advance notice.

• Principle: Human Alternatives, Consideration, and Fallback

The Blueprint summarizes this principle as meaning that “[y]ou should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.”

The bill would implement this principle by requiring a deployer to accommodate a natural person’s request to not be subject to an ADT and to instead be subject to an alternative selection process or accommodation. The bill limits this requirement in two significant ways, however. First, the alternative is only available in cases in which a consequential decision is to be made solely based on the output of an ADT; i.e., one over which there is no human discretion. Second, the alternative must be technically feasible.

5) Other opposition arguments. Several other arguments raised by bill opponents are as follows.

First, the opposition seeks greater clarity as to what constitutes a violation giving rise to liability under the bill. They write:

As currently drafted, it is possible that a violation could constitute not only a failure to complete an impact assessment altogether, but could also be interpreted as any single deficiency within that impact assessment. Even then, it is unknown whether the number of violations is based on that single error, or by any single error multiplied by the number of individuals who were potentially impacted by the use of a particular type of ADT for which an assessment was required or who received an inadequate notice.

In response, the author may wish to further define what constitutes a violation.

Second, the opposition points out that at least two California agencies—the Civil Rights Department and the California Privacy Protection Agency—are currently engaged in rulemaking proceedings with respect to ADTs. The opposition argues:

[I]t is difficult to foresee how such laws and regulations will layer on top of one another and whether there will be conflicting public policy around the use of such tools and technologies. […] Understandably, our members are alarmed by the likelihood of conflict and confusion at the conclusion of these efforts that are being run in parallel to each other, without any coordination or consideration of the other efforts underway.

The author may wish to clarify the boundaries between this bill and any regulations forthcoming from these state agencies, so as to better facilitate compliance.

Finally, opponents believe impact assessments should not be subject to California Public Records Act requests, and that greater restrictions need to be placed on sharing of impact assessments between state agencies. Yet imposing these restrictions may curtail informed future responses to ADTs by both the public and by state government. The author may wish to consider the appropriate balance regarding availability or confidentiality of impact assessments submitted to the Civil Rights Department.

6) Author’s amendments—technical and clarifying changes. Author’s amendments to the bill, reflected in the analysis above, make the following technical and clarifying changes:

22756.1 (a) […]

(5) An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployer’s use of the automated decision tool. […]

(b) […]

(4) An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the use of the automated decision tool.

The two preceding sets of amendments clarify that is the use of the ADT that may give rise to potential adverse impacts.

22756.1 […]

(d) (1) Subject to paragraph (2), t This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year,

(2) This subdivision does not apply if the deployer deployeds an automated decision tool that impacteds more than 999 people per year as of the end of the prior calendar year.

[…]

22756.4 […]

(c) (1) Subject to paragraph (2), t This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year,

(2) This subdivision does not apply if the deployer deployeds an automated decision tool that impacteds more than 999 people per year as of the end of the prior calendar year.

The two preceding sets of amendments are intended to simplify the wording of these paragraphs.

22756.4. […]

(b) […]

(4) If established by a developer, provide for compliance with Sections 22756.1 2 and 22756.3.

This amendment corrects an erroneous cross-reference, as Section 22756.1 applies to deployers, not developers.

22756.5. A deployer or developer shall make publicly available, in a readily accessible manner, a clear artificial intelligence policy that provides a summary of both of the following: […]

Use of the phrase “artificial intelligence policy” may have led some to believe that this section provided a statutory definition of an artificial intelligence policy, which was not the author’s intent.

7) Related legislation. AB 302 (Ward) would require the California Department of Technology (CDT) to conduct an inventory of all high-risk automated decision systems being used in state agencies. Status: Assembly Appropriations Committee.

SB 313 (Dodd) would establish an Office of Artificial Intelligence within CDT, with “the powers and authorities necessary to guide the design, use, or deployment of automated systems by a state agency to ensure that all AI systems are designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and that minimizes bias and promotes equitable outcomes for all Californians.” Status: Senate Governmental Organization Committee.

SB 721 (Becker) would create the California Interagency AI Working Group, which would deliver a report to the Legislature regarding artificial intelligence, include a recommendation for a statutory definition of “artificial intelligence” for use in legislation. Status: Senate Governmental Organization Committee.

AB 13 (Chau, 2021) would have established the Automated Decision Systems (ADS) Accountability Act of 2021. The Act would have required state agencies seeking to procure ADS for high-risk applications to consider, among other things, steps taken by a prospective contractor to identify and mitigate potential disparate impacts that could result from use of that ADS; required a prospective contractor for an ADS for a high-risk application to submit an ADS impact assessment containing specified information about the ADS; and required the contracting agency to submit to CDT a high-risk ADS accountability report containing specified information regarding their proposed use of the ADS. Status: Gut-and-amend for another purpose.

ARGUMENTS IN SUPPORT: The California-Hawaii State Conference of the NAACP emphasizes the importance of this bill for safeguarding civil rights:

ADTs have been found to exhibit biases and consequently have resulted in discriminatory impacts and harm to marginalized communities. A study published in Science showed that a clinical algorithm used across hospitals for determining patient care was racially biased against Black patients. The algorithm used healthcare spending as a proxy for health needs and falsely concluded that Black patients were healthier than equally sick white patients, depriving Black patients from needed high-risk care. These results were biased because Black patients face disproportionate poverty levels and spend less on health care.

ARGUMENTS IN OPPOSITION: The opposition coalition warns of unintended harmful consequences:

[O]verregulation in this space can easily undermine many beneficial uses of ADT—including the ability to develop and deploy these tools in a manner that can in fact reduce the instances and effects of human bias. To that end, we urge greater clarity, precision, and narrowing of the bill to avoid unintended consequences[.]

REGISTERED SUPPORT / OPPOSITION:

Support

California-Hawaii State Conference of the NAACP

Israeli-American Civic Action Network

Oakland Privacy

Santa Monica Democratic Club

Oppose unless Amended

California Association of Realtors

Opposition

American Financial Services Association

California Apartment Association

California Bankers Association

California Chamber of Commerce

California Credit Union League

California Financial Services Association

California Grocers Association

California League of Food Producers

California Manufacturers & Technology Association

California Retailers Association

Card Coalition

Civil Justice Association of California

Computer & Communications Industry Association

Insights Association

National Payroll Reporting Consortium

Netchoice

Software & Information Industry Association

State Privacy and Security Coalition, INC.

TechNet

Analysis Prepared by: Jith Meganathan / P. & C.P. / (916) 319-2200