AB 2930: Automated decision systems.
- Session Year: 2023-2024
- House: Assembly
- Latest Version Date: 2024-08-28
Current Status:
Failed
(2024-08-31: Ordered to inactive file at the request of Senator Umberg.)
Introduced
In Committee
First Chamber
In Committee
Second Chamber
Enacted
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
This bill would, among other things, require, as prescribed, a deployer, as defined, and a developer of an automated decision tool, system, as defined, to perform an impact assessment on any automated decision tool system before the tool system is first deployed and annually thereafter that includes, among other things, a statement of the purpose of the automated decision tool system and its intended benefits, uses, and deployment contexts. The bill would require a deployer or a developer to provide any impact assessment that it performed to the Civil Rights Department and would exempt an impact assessment provided to the department from the California Public Records Act, as prescribed.
This bill would require a deployer to, prior to an automated decision tool system making a consequential decision, as defined, or being a substantial factor, as defined, in making a consequential decision, notify any natural person that is subject to the consequential decision that an automated decision tool system is being used and to provide that person with specified information. The bill would require a deployer that has deployed an automated decision tool system to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide to the natural person, among other things, an opportunity to correct any incorrect personal data. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, system, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool system and to instead be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool system if an impact assessment identifies a reasonable risk of algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool system contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law, until that risk has been mitigated. The bill would also prohibit a developer from making available to potential deployers an automated decision tool system until the risk of algorithmic discrimination has been mitigated.
This bill would authorize the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law, the California Consumer Privacy Act of 2018 (CCPA), grants to a consumer various rights with respect to personal information, as defined, that is collected by a business, as defined, including the right to request that a business delete personal information about the consumer that the business has collected from the consumer. Existing law, the California Privacy Rights Act of 2020, an initiative measure approved by the voters as Proposition 24 at the November 3, 2020, statewide general election, amended, added to, and reenacted the CCPA. The CCPA establishes the California Privacy Protection Agency with full administrative power, authority, and jurisdiction to implement and enforce the CCPA.This bill would, among other things, require, as prescribed, a deployer, as defined, and a developer of an automated decision tool, as defined, to perform an impact assessment on any automated decision tool before the tool is first deployed and annually thereafter that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the California Privacy Protection Agency within 30 days of a request by the agency and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the agency. The bill would exempt an impact assessment from the California Public Records Act, as specified.
This bill would require the California Privacy Protection Agency to, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer, as defined, is required to comply with specified deployer requirements for each deployed automated decision tool. The bill would require full compliance by January 1, 2031. The bill would require a state government deployer to, by January 1, 2026, provide to the agency a list of automated decision tools initially deployed before January 1, 2025.This bill would require a deployer to, prior to an automated decision tool making a consequential decision, as defined, or being a substantial factor, as defined, in making a consequential decision, notify any natural person that is subject to the consequential decision that an automated decision tool is being used and to provide that person with specified information. The bill would require a deployer that has deployed an automated decision tool to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide to the natural person, among other things, an opportunity to correct any incorrect personal data. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to instead be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool if an impact assessment identifies a reasonable risk of algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law, until that risk has been mitigated. The bill would also prohibit a developer from making available to potential deployers an automated decision tool until the risk of algorithmic discrimination has been mitigated.
This bill would authorize certain public attorneys, including the Attorney General, and the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney or the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.This bill would make legislative findings to that effect.This bill would declare that its provisions further the purposes and intent of the California Privacy Rights Act of 2020.The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law, the California Consumer Privacy Act of 2018 (CCPA), grants to a consumer various rights with respect to personal information, as defined, that is collected by a business, as defined, including the right to request that a business delete personal information about the consumer that the business has collected from the consumer. Existing law, the California Privacy Rights Act of 2020, an initiative measure approved by the voters as Proposition 24 at the November 3, 2020, statewide general election, amended, added to, and reenacted the CCPA. The CCPA establishes the California Privacy Protection Agency with full administrative power, authority, and jurisdiction to implement and enforce the CCPA.
This bill would, among other things, require, as prescribed, a deployer, as defined, and a developer of an automated decision tool, as defined, to perform an impact assessment on any automated decision tool before the tool is first deployed and annually thereafter that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the California Privacy Protection Agency within 30 days of a request by the agency and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the agency. The bill would exempt an impact assessment from the California Public Records Act, as specified.
This bill would require the California Privacy Protection Agency to, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer, as defined, is required to comply with specified deployer requirements for each deployed automated decision tool. The bill would require full compliance by January 1, 2031. The bill would require a state government deployer to, by January 1, 2026, provide to the agency a list of automated decision tools initially deployed before January 1, 2025.
This bill would require a deployer to, prior to an automated decision tool making a consequential decision, as defined, or being a substantial factor, as defined, in making a consequential decision, notify any natural person that is subject to the consequential decision that an automated decision tool is being used and to provide that person with specified information. The bill would require a deployer that has deployed an automated decision tool to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide to the natural person, among other things, an opportunity to correct any incorrect personal data. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to instead be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool if an impact assessment identifies a reasonable risk of algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law, until that risk has been mitigated. The bill would also prohibit a developer from making available to potential deployers an automated decision tool until the risk of algorithmic discrimination has been mitigated.
This bill would authorize certain public attorneys, including the Attorney General, and the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney or the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
This bill would declare that its provisions further the purposes and intent of the California Privacy Rights Act of 2020.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
Existing law, the California Consumer Privacy Act of 2018 (CCPA), grants to a consumer various rights with respect to personal information, as defined, that is collected by a business, as defined, including the right to request that a business delete personal information about the consumer that the business has collected from the consumer. Existing law, the California Privacy Rights Act of 2020, an initiative measure approved by the voters as Proposition 24 at the November 3, 2020, statewide general election, amended, added to, and reenacted the CCPA. The CCPA establishes the California Privacy Protection Agency with full administrative power, authority, and jurisdiction to implement and enforce the CCPA.
This bill would, among other things, require, as prescribed, a deployer, as defined, and a developer of an automated decision tool, as defined, to perform an impact assessment on any automated decision tool before first using it the tool is first deployed and annually thereafter that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 7 California Privacy Protection Agency within 30 days of a request by the department agency and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. agency. The bill would, in complying with a request for public records, require the Civil Rights Department, or an entity with which an impact assessment was shared, to redact any trade secret from the impact assessment. would exempt an impact assessment from the California Public Records Act, as specified.
This bill would require the Civil Rights Department California Privacy Protection Agency to, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer, as defined, is required to comply with specified deployer requirements for each deployed automated decision tool. The bill would require full compliance by January 1, 2031. The bill would require a state government deployer to, by January 1, 2026, provide to the Civil Rights Department agency a list of automated decision tools initially deployed before January 1, 2025.
This bill would require a deployer to, at or before the time prior to an automated decision tool is used to make making a consequential decision, as defined, or being a substantial factor, as defined, in making a consequential decision, notify any natural person that is the subject of to the consequential decision that an automated decision tool is being used and to provide that person with, among other things, a statement of the purpose of the automated decision tool. with specified information. The bill would require a deployer that has deployed an automated decision tool to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide to the natural person, among other things, an opportunity to correct any incorrect personal data. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to instead be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool in a manner that results in if an impact assessment identifies a reasonable risk of algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law. or federal law, until that risk has been mitigated. The bill would also prohibit a developer from making available to potential deployers an automated decision tool that results in until the risk of algorithmic discrimination. discrimination has been mitigated.
This bill would authorize certain public attorneys, including the Attorney General, and the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney or the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
This bill would declare that its provisions further the purposes and intent of the California Privacy Rights Act of 2020.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
This bill would, among other things, require require, as prescribed, a deployer, as defined, and a developer of an automated decision tool, as defined, to perform an impact assessment on any automated decision tool before first using it and annually thereafter, and with respect to an automated decision tool that a deployer first used before January 1, 2025, the bill would require the deployer to perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter, thereafter that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 7 days of a request by the department and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, in complying with a request for public records, require the Civil Rights Department, or an entity with which an impact assessment was shared, to redact any trade secret from the impact assessment.
This bill would require the Civil Rights Department to, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer, as defined, shall is required to comply with specified deployer requirements for each deployed automated decision tool. The bill would require full compliance by January 1, 2031. The bill would require a state government deployer to, by January 1, 2026, provide to the Civil Rights Department a list of automated decision tools initially deployed prior to before January 1, 2025.
This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool in a manner that results in algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law. The bill would also prohibit a developer from making available to potential deployers an automated decision tool that results in algorithmic discrimination.
This bill would authorize certain public attorneys, including the Attorney General, and the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney or the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to perform an impact assessment on any automated decision tool before first using it and annually thereafter, and with respect to an automated decision tool that a deployer first used before January 1, 2025, the bill would require the deployer to perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter, that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 7 days of a request by the department and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, in complying with a request for public records, require the Civil Rights Department, or an entity with which an impact assessment was shared, to redact any trade secret from the impact assessment.
This bill would require the Civil Rights Department to, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer, as defined, shall comply with specified deployer requirements for each deployed automated decision tool. The bill would require full compliance by January 1, 2031. The bill would require a state government deployer to, by January 1, 2026, provide the Civil Rights Department a list of automated decision tools initially deployed prior to January 1, 2025.
This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool in a manner that results in algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law. The bill would also prohibit a developer from making available to potential deployers an automated decision tool that results in algorithmic discrimination.
This bill would authorize certain public attorneys, including the Attorney General, and the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney or the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to perform an impact assessment on any automated decision tool before first using it and annually thereafter, and with respect to an automated decision tool that a deployer first used before January 1, 2025, the bill would require the deployer to perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter, that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 7 days of a request by the department and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, in complying with a request for public records, require the Civil Rights Department, or an entity with which an impact assessment was shared, to redact any trade secret from the impact assessment.
This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool in a manner that results in algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law. The bill would also prohibit a developer from making available to potential deployers an automated decision tool that results in algorithmic discrimination.
This bill would authorize certain public attorneys, including the Attorney General, and the Civil Rights Department to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney or the Civil Rights Department to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to, on or to perform an impact assessment on any automated decision tool before first using it and annually thereafter, and with respect to an automated decision tool that a deployer first used before January 1, 2025, the bill would require the deployer to perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 7 days of a request by the department and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, in complying with a request for public records, require the Civil Rights Department, or an entity with which an impact assessment was shared, to redact any trade secret from the impact assessment.
This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool in a manner that results in algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law. The bill would also prohibit a developer from making available to potential deployers an automated decision tool that results in algorithmic discrimination.
This bill would authorize certain public attorneys, including the Attorney General, to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
The Unruh Civil Rights Act provides that all persons within the jurisdiction of this state are free and equal and, regardless of their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever.
The California Fair Employment and Housing Act establishes the Civil Rights Department within the Business, Consumer Services, and Housing Agency and requires the department to, among other things, bring civil actions to enforce the act.
This bill would, among other things, require a deployer, as defined, and a developer of an automated decision tool, as defined, to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The bill would require a deployer or developer to provide the impact assessment to the Civil Rights Department within 7 days of a request by the department and would punish a violation of that provision with an administrative fine of not more than $10,000 to be recovered in an administrative enforcement action brought by the Civil Rights Department. The bill would, in complying with a request for public records, require the Civil Rights Department, or an entity with which an impact assessment was shared, to redact any trade secret from the impact assessment.
This bill would require a deployer to, at or before the time an automated decision tool is used to make a consequential decision, as defined, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. The bill would, if a consequential decision is made solely based on the output of an automated decision tool, require a deployer to, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation, as prescribed.
This bill would prohibit a deployer from using an automated decision tool in a manner that results in algorithmic discrimination, which the bill would define to mean the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
This bill would authorize certain public attorneys, including the Attorney General, to bring a civil action against a deployer or developer for a violation of the bill and would authorize a court to award, only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation. The bill would require a public attorney to, before commencing an action for injunctive relief, provide 45 days written notice to a deployer or developer of the alleged violations of the bill and would provide a deployer or developer a specified opportunity to cure those violations, if the deployer or developer provides the person who gave the notice an express written statement, under penalty of perjury, that the violation has been cured and that no further violations shall occur. By expanding the scope of the crime of perjury, this bill would impose a state-mandated local program.
Existing constitutional provisions require that a statute that limits the right of access to the meetings of public bodies or the writings of public officials and agencies be adopted with findings demonstrating the interest protected by the limitation and the need for protecting that interest.
This bill would make legislative findings to that effect.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool system contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.(c)(1)Automated decision system means, consistent with Section 11546.45.5 of the Government Code, a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and that is used to make, or be a substantial factor in making, a consequential decision.
(2)Automated decision system does not mean a spam email filter, firewall, antivirus software, identity and access management tool, calculator, database, dataset, or other compilation of data.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, employment with respect to all of the following:
(1)Pay or promotion.
(2)Hiring or termination.
(3)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(e)Deployer means a person, partnership, developer, corporation, or any contractor or agent of those entities, that uses an automated decision tool system to make a consequential decision.
(f)Developer means a person, partnership, or corporation that designs, codes, or produces an automated decision tool, system, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool system that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)Substantial factor means an element of a decisionmaking process that is capable of altering the outcome of the process.
(j)Substantial modification means a new version, new release, or other update to an automated decision tool system that materially changes its uses, intended uses, or outcomes.
(k)Unlawful discrimination means any act that violates Section 51 of the Civil Code, any act that constitutes an unlawful practice or unlawful employment practice under Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code, or any other practice or act that otherwise violates a state or federal law against discrimination.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool system before the tool system is first deployed and annually thereafter.
(2)(A)With respect to an automated decision tool system that a deployer first used prior to January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool system before January 1, 2026, and annually thereafter.
(B)This subdivision does not require a deployer to perform an impact assessment on an automated decision tool system before using it if all of the following are true:
(i)The deployer uses the automated decision tool system only for its intended use as determined by the developer of the automated decision tool. system.
(ii)The deployer does not make any substantial modifications to the automated decision tool. system.
(iii)The developer of the automated decision tool system has performed any impact assessment on the automated decision tool system required by subdivision (c).
(iv)The developer of the automated decision tool system has provided documentation to the deployer pursuant to Section 22756.3.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool system and its intended benefits, uses, and deployment contexts.
(2)A description of all of the following:
(A)The personal characteristics or attributes that the automated decision tool system will measure or assess.
(B)The method by which the automated decision tool system measures or assesses those attributes or characteristics.
(C)How those attributes or characteristics are relevant to the consequential decisions for which the automated decision tool system will be used.
(D)The automated decision tools systems outputs.
(E)How outputs are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool system when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool system is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool. system.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool. system. The description must shall address all of the following:
(A)Whether the automated decision tool system could be modified to mitigate the risk of algorithmic discrimination.
(B)Whether effective accommodations can be provided for any limitations on accessibility.
(C)Whether less discriminatory procedures or methods could be employed to mitigate the risk of algorithmic discrimination.
(7)A description of how the automated decision tool system will be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool system has been or will be evaluated for validity, reliability, and relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool system that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool system and annually thereafter.
(2)With respect to an automated decision tool system that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool system before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool system and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools systems outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool system when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool. system.
(5)A description of the measures taken by the developer to mitigate the risk of algorithmic discrimination arising from the use of the automated decision tool. system.
(6)A description of how the automated decision tool system can be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
(7)A description of how the automated decision tool system has been evaluated for validity, reliability, and relevance.
(e)A deployer or developer shall perform, as soon as feasible, an impact assessment with respect to any substantial modification to an automated decision tool. system.
(f)This section does not apply to a deployer with fewer than 55 employees unless the deployer used an automated decision tool system that impacted more than 999 people during the previous calendar year.
22756.2.(a)(1)Prior to an automated decision tool system making a consequential decision, or being a substantial factor in making a consequential decision, a deployer shall notify any natural person that is subject to the consequential decision that an automated decision tool system is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool. system.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool system that includes all of the following:
(i)The personal characteristics or attributes that the automated decision tool system will measure or assess.
(ii)The method by which the automated decision tool system measures or assesses those attributes or characteristics.
(iii)How those attributes or characteristics contribute to the consequential decision.
(iv)The format and structure of the automated decision tools systems outputs.
(v)How those outputs are used to make, be a substantial factor in making, a consequential decision.
(vi)A summary of the most recent impact assessment performed on the automated decision tool. system.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, system, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, system, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool system and to instead be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
(c)A deployer that has deployed an automated decision tool, system, to make, or be a substantial factor in making, a consequential decision concerning a natural person, shall provide to the natural person all of the following:
(1)A simple and actionable explanation that identifies the principal factors, characteristics, logic, and other information related to the individual that led to the consequential decision.
(2)The role that the automated decision tool system played in the decisionmaking process.
(3)The opportunity to correct any incorrect personal data that the automated decision tool system processed in making, or as a substantial factor in making, the consequential decision.
(d)All notices and other communications described in this section shall be all of the following:
(1)Transmitted directly to the subject of the consequential decision when possible, or else made available in a manner reasonably calculated to ensure that the subjects of consequential decisions receive actual notice.
(2)Provided in English, in any non-English language spoken by at least 1 percent of the population of this state as of the most recent United States Census, and in any other language that the deployer regularly uses to communicate with the subjects of consequential decisions.
(3)Written in clear and plain language.
(4)Made available in formats that are accessible to people who are blind or have other disabilities.
(5)Otherwise presented in a manner that ensures the communication clearly and effectively conveys the required information to subjects of the relevant consequential decisions.
22756.3.(a)A developer shall provide a deployer with the results of any impact assessment performed on an automated decision tool system that the developer sells, licenses, or otherwise transfers to the deployer, along with documentation describing all of the following:
(1)The intended uses of the automated decision tool. system.
(2)The known limitations of the automated decision tool, system, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(3)The type of data used to program or train the automated decision tool. system.
(4)How the automated decision tool system was evaluated for validity and explainability before sale or licensing.
(5)The deployers responsibilities under this chapter.
(6)Any technical information necessary for a deployer to fulfill their obligations under Section 22756.2.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code. To the extent that a developer withholds information pursuant to this section, the developer shall notify the deployer and provide a basis for the withholding.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool. system.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool. system.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool. system.
(E)The technical feasibility and cost of available tools, systems, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool. system.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool system fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool. system.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, system, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 55 employees unless the deployer used an automated decision tool system that impacted more than 999 people during the previous calendar year.
22756.5.A deployer and developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools systems currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools systems it currently uses or makes available to others.
22756.6.(a)If an impact assessment performed by a deployer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the automated decision tool system until the risk has been mitigated.
(b)If an impact assessment performed by a developer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination under deployment conditions reasonably likely to occur in this state, the developer shall not make the automated decision tool system available to potential deployers until the risk has been mitigated.
22756.7.(a)The Civil Rights Department may investigate a possible violation of this chapter and may request an impact assessment performed pursuant to this chapter in order to carry out the investigation.
(b)(1)Within 30 days of receiving a request from the Civil Rights Department, a deployer or a developer shall provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)An impact assessment disclosed to the Civil Rights Department pursuant to this chapter shall be exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
22756.9.(a)The Civil Rights Department may bring a civil action against a deployer or developer for a violation of this chapter: chapter.
(b)A court may award in an action brought pursuant to this section all of the following:
(1)Injunctive relief.
(2)Declaratory relief.
(3)Reasonable attorneys fees and litigation costs.
(4)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(c)(1)The Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.10.It shall be unlawful for a deployer or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.11.This chapter does not apply to cybersecurity-related technology, including technology designed to detect, protect against, or respond to security incidents, identity theft, fraud, harassment, malicious or deceptive activities or any illegal activity, preserve the integrity or security of systems, or investigate, report, or prosecute those responsible for those actions.
22756.12.(a)The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
(b)This chapter does not diminish the rights, privileges, or remedies of an employee under any other federal or state law or under any employment contract or collective bargaining agreement.
(c)This chapter does not authorize any use of automated decision systems that may be limited, restricted, or prohibited under any other applicable law.
(d)(1)This chapter does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
(2)If a developer or deployer withholds information pursuant to this subdivision, the developer or deployer shall notify the relevant entity or natural person and provide a basis for the withholding.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
SEC. 2.SEC. 3.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, any of the following: employment with respect to all of the following:
(1)Employment with respect to all of the following:(A)(1)Pay or promotion.
(B)(2)Hiring or termination.
(C)(3)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:(A)Assessment or placement.(B)Detecting student cheating or plagiarism.(C)Accreditation.(D)Certification.(E)Admissions or enrollment.(F)Discipline.(G)Evaluation. (H)Financial aid or scholarships.(3)Housing or lodging, including rental or short-term housing or lodging.(4)All of the following essential utilities:(A)Electricity.(B)Heat.(C)Water.(D)Internet or telecommunications access.(E)Transportation.(5)Family planning.(6)Adoption services, reproductive services, or assessments related to child protective services.(7)Health care or health insurance, including mental health care, dental, or vision.(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.(9)All of the following aspects of the criminal justice system:(A)Risk assessments for pretrial hearings.(B)Sentencing.(C)Parole.(10)Legal services.(11)Private arbitration.(12)Mediation.(13)Voting.(e)Deployer means a person, partnership, local government agency, developer, corporation, or any contractor or agent of those entities, that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)State government deployer means a state government agency that uses an automated decision tool to make a consequential decision.(j)(i)Substantial factor means an element of a decisionmaking process that is capable of altering the outcome of the process.
(k)(j)Substantial modification means a new version, new release, or other update to an automated decision tool that materially changes its uses, intended uses, or outcomes.
(l)(k)Unlawful discrimination means any act that violates Section 51 of the Civil Code, any act that constitutes an unlawful practice or unlawful employment practice under Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code, or any other practice or act that otherwise violates a state or federal law against discrimination.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before the tool is first deployed and annually thereafter.
(2)(A)With respect to an automated decision tool that a deployer first used prior to January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(B)This subdivision does not require a deployer to perform an impact assessment on an automated decision tool before using it if all of the following are true:
(i)The deployer uses the automated decision tool only for its intended use as determined by the developer of the automated decision tool.
(ii)The deployer does not make any substantial modifications to the automated decision tool.
(iii)The developer of the automated decision tool has performed any impact assessment on the automated decision tool required by subdivision (c).
(iv)The developer of the automated decision tool has provided documentation to the deployer pursuant to Section 22756.3.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of all of the following:
(A)The personal characteristics or attributes that the automated decision tool will measure or assess.
(B)The method by which the automated decision tool measures or assesses those attributes or characteristics.
(C)How those attributes or characteristics are relevant to the consequential decisions for which the automated decision tool will be used.
(D)The automated decision tools outputs.
(E)How outputs are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool. The description must address all of the following:
(A)Whether the automated decision tool could be modified to mitigate the risk of algorithmic discrimination.
(B)Whether effective accommodations can be provided for any limitations on accessibility.
(C)Whether less discriminatory procedures or methods could be employed to mitigate the risk of algorithmic discrimination.
(7)A description of how the automated decision tool will be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity, reliability, and relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
(7)A description of how the automated decision tool has been evaluated for validity, reliability, and relevance.
(e)A deployer or developer shall perform, as soon as feasible, an impact assessment with respect to any substantial modification to an automated decision tool.
(f)This section does not apply to a deployer with fewer than 55 employees unless the deployer used an automated decision tool that impacted more than 999 people during the previous calendar year.
22756.2.(a)(1)Prior to an automated decision tool making a consequential decision, or being a substantial factor in making a consequential decision, a deployer shall notify any natural person that is subject to the consequential decision that an automated decision tool is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes all of the following:
(i)The personal characteristics or attributes that the automated decision tool will measure or assess.
(ii)The method by which the automated decision tool measures or assesses those attributes or characteristics.
(iii)How those attributes or characteristics contribute to the consequential decision.
(iv)The format and structure of the automated decision tools outputs.
(v)How those outputs are used to make, be a substantial factor in making, a consequential decision.
(vi)A summary of the most recent impact assessment performed on the automated decision tool.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to instead be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
(c)A deployer that has deployed an automated decision tool, to make, or be a substantial factor in making, a consequential decision concerning a natural person, shall provide to the natural person all of the following:
(1)A simple and actionable explanation that identifies the principal factors, characteristics, logic, and other information related to the individual that led to the consequential decision.
(2)The role that the automated decision tool played in the decisionmaking process.
(3)The opportunity to correct any incorrect personal data that the automated decision tool processed in making, or as a substantial factor in making, the consequential decision.
(d)All notices and other communications described in this section shall be all of the following:
(1)Transmitted directly to the subject of the consequential decision when possible, or else made available in a manner reasonably calculated to ensure that the subjects of consequential decisions receive actual notice.
(2)Provided in English, in any non-English language spoken by at least 1 percent of the population of this state as of the most recent United States Census, and in any other language that the deployer regularly uses to communicate with the subjects of consequential decisions.
(3)Written in clear and plain language.
(4)Made available in formats that are accessible to people who are blind or have other disabilities.
(5)Otherwise presented in a manner that ensures the communication clearly and effectively conveys the required information to subjects of the relevant consequential decisions.
22756.3.(a)A developer shall provide a deployer with the results of any impact assessment performed on an automated decision tool that the developer sells, licenses, or otherwise transfers to the deployer, along with documentation describing all of the following:
(1)The intended uses of the automated decision tool.
(2)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(3)The type of data used to program or train the automated decision tool.
(4)How the automated decision tool was evaluated for validity and explainability before sale or licensing.
(5)The deployers responsibilities under this chapter.
(6)Any technical information necessary for a deployer to fulfill their obligations under Section 22756.2.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code. To the extent that a developer withholds information pursuant to this section, the developer shall notify the deployer and provide a basis for the withholding.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 55 employees unless the deployer used an automated decision tool that impacted more than 999 people during the previous calendar year.
22756.5.A deployer and developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)If an impact assessment performed by a deployer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the automated decision tool until the risk has been mitigated.
(b)If an impact assessment performed by a developer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination under deployment conditions reasonably likely to occur in this state, the developer shall not make the automated decision tool available to potential deployers until the risk has been mitigated.
22756.7.(a)A state government deployer shall, by January 1, 2026, provide the California Privacy Protection Agency a list of automated decision tools initially deployed prior to January 1, 2025, by the state government deployer. The list shall identify all of the following:(1)Each automated decision tool deployed by the state government deployer.(2)The role of each automated decision tool in making consequential decisions.(3)The population affected by each automated decision tool.(b)The California Privacy Protection Agency shall, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer shall comply with Sections 22756.1, 22756.2, 22756.3, 22756.4, 22756.5, and 22756.6 for each automated decision tool deployed by the state government deployer.(1)The schedule established by the agency shall prioritize compliance for automated decision tools used by state government deployers with the highest risk for algorithmic discrimination, including civil rights violations and other discriminatory outcomes.(2)The schedule established by the agency shall require full compliance by each state government deployer by January 1, 2031.(3)A state government deployer that fails to comply with the requirements of this chapter as specified in the schedule established by the agency may be subject to enforcement by the agency, as authorized in Sections 22756.8 and 22756.9.22756.8.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.(b)(1)Upon receiving a request from the California Privacy Protection Agency, a deployer or a developer shall, within 30 days of the request, provide any impact assessment that it performed pursuant to this chapter to the California Privacy Protection Agency.(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.(3)(A)An impact assessment disclosed to the California Privacy Protection Agency pursuant to this chapter, or that is subsequently shared by the California Privacy Protection Agency with another agency or department, shall be exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).(B)No provision of this chapter shall be construed to require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code. To the extent that a developer or deployer withholds information pursuant to this chapter, the developer or deployer shall notify the relevant entity or natural person and provide a basis for the withholding.(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the California Privacy Protection Agency.(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.(c)The California Privacy Protection Agency may provide an impact assessment it receives to any of the following public entities to assist that entity in initiating or litigating a civil action:(1)The Attorney General in the name of the people of the State of California.(2)A district attorney, county counsel, or city attorney.(3)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.(4)The Civil Rights Department.22756.9.(a)(1)Any of the following public entities The Civil Rights Department may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.(D)The Civil Rights Department.(2)(b)A court may award in an action brought pursuant to this subdivision section all of the following:
(A)(1)Injunctive relief.
(B)(2)Declaratory relief.
(C)(3)Reasonable attorneys fees and litigation costs.
(D)(4)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(c)(1)A public attorney, or the The Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.10.It shall be unlawful for a deployer, state government deployer, deployer or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.11.This chapter does not apply to cybersecurity-related technology, including technology designed to detect, protect against, or respond to security incidents, identity theft, fraud, harassment, malicious or deceptive activities or any illegal activity, preserve the integrity or security of systems, or investigate, report, or prosecute those responsible for those actions.
22756.12.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.SEC. 3.The Legislature finds and declares that this act furthers the purposes and intent of the California Privacy Rights Act of 2020.SEC. 4.SEC. 2.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment or placement.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions or enrollment.
(F)Discipline.
(G)Evaluation.
(H)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, local government agency, developer, corporation, or any contractor or agent of those entities, that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)State government deployer means a state government agency that uses an automated decision tool to make a consequential decision.
(j)Substantial factor means an element of a decisionmaking process that is capable of altering the outcome of the process.
(k)Substantial modification means a new version, new release, or other update to an automated decision tool that materially changes its uses, intended uses, or outcomes.
(l)Unlawful discrimination means any act that violates Section 51 of the Civil Code, any act that constitutes an unlawful practice or unlawful employment practice under Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code, or any other practice or act that otherwise violates a state or federal law against discrimination.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before the tool is first deployed and annually thereafter.
(2)(A)With respect to an automated decision tool that a deployer first used prior to January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(B)This subdivision does not require a deployer to perform an impact assessment on an automated decision tool before using it if all of the following are true:
(i)The deployer uses the automated decision tool only for its intended use as determined by the developer of the automated decision tool.
(ii)The deployer does not make any substantial modifications to the automated decision tool.
(iii)The developer of the automated decision tool has performed any impact assessment on the automated decision tool required by subdivision (c).
(iv)The developer of the automated decision tool has provided documentation to the deployer pursuant to Section 22756.3.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of all of the following:
(A)The personal characteristics or attributes that the automated decision tool will measure or assess.
(B)The method by which the automated decision tool measures or assesses those attributes or characteristics.
(C)How those attributes or characteristics are relevant to the consequential decisions for which the automated decision tool will be used.
(D)The automated decision tools outputs.
(E)How outputs are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool. The description must address all of the following:
(A)Whether the automated decision tool could be modified to mitigate the risk of algorithmic discrimination.
(B)Whether effective accommodations can be provided for any limitations on accessibility.
(C)Whether less discriminatory procedures or methods could be employed to mitigate the risk of algorithmic discrimination.
(7)A description of how the automated decision tool will be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity, reliability, and relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
(7)A description of how the automated decision tool has been evaluated for validity, reliability, and relevance.
(e)A deployer or developer shall perform, as soon as feasible, an impact assessment with respect to any substantial modification to an automated decision tool.
(f)This section does not apply to a deployer with fewer than 55 employees unless the deployer used an automated decision tool that impacted more than 999 people during the previous calendar year.
22756.2.(a)(1)Prior to an automated decision tool making a consequential decision, or being a substantial factor in making a consequential decision, a deployer shall notify any natural person that is subject to the consequential decision that an automated decision tool is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes all of the following:
(i)The personal characteristics or attributes that the automated decision tool will measure or assess.
(ii)The method by which the automated decision tool measures or assesses those attributes or characteristics.
(iii)How those attributes or characteristics contribute to the consequential decision.
(iv)The format and structure of the automated decision tools outputs.
(v)How those outputs are used to make, be a substantial factor in making, a consequential decision.
(vi)A summary of the most recent impact assessment performed on the automated decision tool.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to instead be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
(c)A deployer that has deployed an automated decision tool, to make, or be a substantial factor in making, a consequential decision concerning a natural person, the deployer shall provide to the natural person all of the following:
(1)A simple and actionable explanation that identifies the principal factors, characteristics, logic logic, and other information related to the individual that led to the consequential decision.
(2)The role that the automated decision tool played in the decisionmaking process.
(3)The opportunity to correct any incorrect personal data that the automated decision tool processed in making, or as a substantial factor in making, the consequential decision.
(d)All notices and other communications described in this section shall be all of the following:
(1)Transmitted directly to the subject of the consequential decision when possible, or else made available in a manner reasonably calculated to ensure that the subjects of consequential decisions receive actual notice.
(2)Provided in English, in any non-English language spoken by at least 1 percent of the population of this state as of the most recent United States Census, and in any other language that the deployer regularly uses to communicate with the subjects of consequential decisions.
(3)Written in clear and plain language.
(4)Made available in formats that are accessible to people who are blind or have other disabilities.
(5)Otherwise presented in a manner that ensures the communication clearly and effectively conveys the required information to subjects of the relevant consequential decisions.
22756.3.(a)A developer shall provide a deployer with the results of any impact assessment performed on an automated decision tool that the developer sells, licenses, or otherwise transfers to the deployer, along with documentation describing all of the following:
(1)The intended uses of the automated decision tool.
(2)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(3)The type of data used to program or train the automated decision tool.
(4)How the automated decision tool was evaluated for validity and explainability before sale or licensing.
(5)The deployers responsibilities under this chapter.
(6)Any technical information necessary for a deployer to fulfill their obligations under Section 22756.2.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code. To the extent that a developer withholds information pursuant to this section, the developer shall notify the deployer and provide a basis for the withholding.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 55 employees unless the deployer used an automated decision tool that impacted more than 999 people during the previous calendar year.
22756.5.A deployer and developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)If an impact assessment performed by a deployer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the automated decision tool until the risk has been mitigated.
(b)If an impact assessment performed by a developer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination under deployment conditions reasonably likely to occur in this state, the developer shall not make the automated decision tool available to potential deployers until the risk has been mitigated.
22756.7.(a)A state government deployer shall, by January 1, 2026, provide the California Privacy Protection Agency a list of automated decision tools initially deployed prior to January 1, 2025, by the state government deployer. The list shall identify all of the following:
(1)Each automated decision tool deployed by the state government deployer.
(2)The role of each automated decision tool in making consequential decisions.
(3)The population affected by each automated decision tool.
(b)The California Privacy Protection Agency shall, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer shall comply with Sections 22756.1, 22756.2, 22756.3, 22756.4, 22756.5, and 22756.6 for each automated decision tool deployed by the state government deployer.
(1)The schedule established by the agency shall prioritize compliance for automated decision tools used by state government deployers with the highest risk for algorithmic discrimination, including civil rights violations and other discriminatory outcomes.
(2)The schedule established by the agency shall require full compliance by each state government deployer by January 1, 2031.
(3)A state government deployer that fails to comply with the requirements of this chapter as specified in the schedule established by the agency may be subject to enforcement by the agency, as authorized in Sections 22756.8 and 22756.9.
22756.8.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon receiving a request from the California Privacy Protection Agency, a deployer or a developer shall, within 30 days of the request, provide any impact assessment that it performed pursuant to this chapter to the California Privacy Protection Agency.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)An impact assessment disclosed to the California Privacy Protection Agency pursuant to this chapter, or that is subsequently shared by the California Privacy Protection Agency with another agency or department, shall be exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)No provision of this chapter shall be construed to require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code. To the extent that a developer or deployer withholds information pursuant to this chapter, the developer or deployer shall notify the relevant entity or natural person and provide a basis for the withholding.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the California Privacy Protection Agency.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
(c)The California Privacy Protection Agency may provide an impact assessment it receives to any of the following public entities to assist that entity in initiating or litigating a civil action:
(1)The Attorney General in the name of the people of the State of California.
(2)A district attorney, county counsel, or city attorney.
(3)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(4)The Civil Rights Department.
22756.9.(a)(1)Any of the following public entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D)The Civil Rights Department.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, or the Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.10.It shall be unlawful for a deployer, state government deployer, or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.11.This chapter does not apply to cybersecurity-related technology, including technology designed to detect, protect against, or respond to security incidents, identity theft, fraud, harassment, malicious or deceptive activities or any illegal activity, preserve the integrity or security of systems, or investigate, report, or prosecute those responsible for those actions.
22756.12.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to agencies and departments pursuant to Section 1 of this act remain confidential.
SEC. 3.The Legislature finds and declares that this act furthers the purposes and intent of the California Privacy Rights Act of 2020.
SEC. 4.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unjustified unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means a system or service that uses artificial intelligence and has been specifically developed to, or specifically modified to, make, or be a substantial factor in making, consequential decisions. an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment.(A)Assessment or placement.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions.(E)Admissions or enrollment.
(F)Discipline.
(G)Evaluation.
(F)(H)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, local government agency, developer, or corporation corporation, or any contractor or agent of those entities, that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)Significant update means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or outcome.(j)(i)State government deployer means a state government agency that uses an automated decision tool to make a consequential decision.
(j)Substantial factor means an element of a decisionmaking process that is capable of altering the outcome of the process.
(k)Substantial modification means a new version, new release, or other update to an automated decision tool that materially changes its uses, intended uses, or outcomes.
(l)Unlawful discrimination means any act that violates Section 51 of the Civil Code, any act that constitutes an unlawful practice or unlawful employment practice under Part 2.8 (commencing with Section 12900) of Division 3 of Title 2 of the Government Code, or any other practice or act that otherwise violates a state or federal law against discrimination.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before first using it the tool is first deployed and annually thereafter.
(2)(A)With respect to an automated decision tool that a deployer first used before prior to January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(B)This subdivision does not require a deployer to perform an impact assessment on an automated decision tool before using it if all of the following are true:
(i)The deployer uses the automated decision tool only for the its intended use as determined by the developer of the automated decision tool.
(ii)The deployer does not make any significant updates substantial modifications to the automated decision tool.
(iii)The developer of the automated decision tool has performed any impact assessment on the automated decision tool required by subdivision (c).
(iv)The developer of the automated decision tool has provided documentation to the deployer pursuant to Section 22756.3.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the all of the following:
(A)The personal characteristics or attributes that the automated decision tool will measure or assess.
(B)The method by which the automated decision tool measures or assesses those attributes or characteristics.
(C)How those attributes or characteristics are relevant to the consequential decisions for which the automated decision tool will be used.
(D)The automated decision tools outputs and how they outputs.
(E)How outputs are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of potential the risk of algorithmic discrimination, including adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment. tool. The description must address all of the following:
(A)Whether the automated decision tool could be modified to mitigate the risk of algorithmic discrimination.
(B)Whether effective accommodations can be provided for any limitations on accessibility.
(C)Whether less discriminatory procedures or methods could be employed to mitigate the risk of algorithmic discrimination.
(7)A description of how the automated decision tool will be used by a natural person, or be monitored when it is used, used autonomously, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity or validity, reliability, and relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of a potential the risk of algorithmic discrimination, including adverse impact impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information information, or any other classification protected by state or federal law, resulting from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or be monitored when it is used, used autonomously, to make, or be a substantial factor in making, a consequential decision.
(7)A description of how the automated decision tool has been evaluated for validity, reliability, and relevance.
(e)A deployer or developer shall perform, as soon as feasible, an impact assessment with respect to any significant update. substantial modification to an automated decision tool.
(f)This section does not apply to a deployer with fewer than 25 55 employees unless, as of the end of the prior calendar year, unless the deployer deployed used an automated decision tool that impacted more than 999 people per during the previous calendar year.
22756.2.(a)(1)A deployer shall, at or before the time Prior to an automated decision tool is used to make making a consequential decision, or being a substantial factor in making a consequential decision, a deployer shall notify any natural person that is the subject of to the consequential decision that an automated decision tool is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision. all of the following:
(i)The personal characteristics or attributes that the automated decision tool will measure or assess.
(ii)The method by which the automated decision tool measures or assesses those attributes or characteristics.
(iii)How those attributes or characteristics contribute to the consequential decision.
(iv)The format and structure of the automated decision tools outputs.
(v)How those outputs are used to make, be a substantial factor in making, a consequential decision.
(vi)A summary of the most recent impact assessment performed on the automated decision tool.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to instead be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
(c)A deployer that has deployed an automated decision tool, to make, or be a substantial factor in making, a consequential decision concerning a natural person, the deployer shall provide to the natural person all of the following:
(1)A simple and actionable explanation that identifies the principal factors, characteristics, logic and other information related to the individual that led to the consequential decision.
(2)The role that the automated decision tool played in the decisionmaking process.
(3)The opportunity to correct any incorrect personal data that the automated decision tool processed in making, or as a substantial factor in making, the consequential decision.
(d)All notices and other communications described in this section shall be all of the following:
(1)Transmitted directly to the subject of the consequential decision when possible, or else made available in a manner reasonably calculated to ensure that the subjects of consequential decisions receive actual notice.
(2)Provided in English, in any non-English language spoken by at least 1 percent of the population of this state as of the most recent United States Census, and in any other language that the deployer regularly uses to communicate with the subjects of consequential decisions.
(3)Written in clear and plain language.
(4)Made available in formats that are accessible to people who are blind or have other disabilities.
(5)Otherwise presented in a manner that ensures the communication clearly and effectively conveys the required information to subjects of the relevant consequential decisions.
22756.3.(a)A developer shall provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following: the results of any impact assessment performed on an automated decision tool that the developer sells, licenses, or otherwise transfers to the deployer, along with documentation describing all of the following:
(1)The intended uses of the automated decision tool.
(1)(2)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(2)A description of the(3)The type of data used to program or train the automated decision tool.
(3)A description of how(4)How the automated decision tool was evaluated for validity and explainability before sale or licensing.
(4)A description of the(5)The deployers responsibilities under this chapter.
(6)Any technical information necessary for a deployer to fulfill their obligations under Section 22756.2.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, manage, and govern and manage the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 25 55 employees unless, as of the end of the prior calendar year, unless the deployer deployed used an automated decision tool that impacted more than 999 people per during the previous calendar year.
22756.5.A deployer or and developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)A deployer shall not use an automated decision tool that results in algorithmic discrimination.(b)A developer shall not make available to potential deployers an automated decision tool that results in algorithmic discrimination.22756.6.(a)If an impact assessment performed by a deployer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the automated decision tool until the risk has been mitigated.
(b)If an impact assessment performed by a developer pursuant to Section 22756.1 identifies a reasonable risk of algorithmic discrimination under deployment conditions reasonably likely to occur in this state, the developer shall not make the automated decision tool available to potential deployers until the risk has been mitigated.
22756.7.(a)A state government deployer shall, by January 1, 2026, provide the Civil Rights Department California Privacy Protection Agency a list of automated decision tools initially deployed prior to January 1, 2025, by the state government deployer. The list shall identify all of the following:
(1)Each automated decision tool deployed by the state government deployer.
(2)The role of each automated decision tool in making consequential decisions.
(3)The population affected by each automated decision tool.
(b)The Civil Rights Department California Privacy Protection Agency shall, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer shall comply with Sections 22756.1, 22756.2, 22756.3, 22756.4, 22756.5, and 22756.6 for each automated decision tool deployed by the state government deployer.
(1)The schedule established by the department agency shall prioritize compliance for automated decision tools used by state government deployers with the highest risk for adverse impacts, algorithmic discrimination, including civil rights violations and other discriminatory outcomes.
(2)The schedule established by the department agency shall require full compliance by each state government deployer by January 1, 2031.
(3)A state government deployer that fails to comply with the requirements of this chapter as specified in the schedule established by the department agency may be subject to enforcement by the department, agency, as authorized in Sections 22756.8 and 22756.9.
(c)The Civil Rights Department may adopt rules and regulations pursuant to the rulemaking provisions of the Administrative Procedure Act (Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code) as necessary to effectuate the intent of this section.22756.8.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon the request of the Civil Rights Department, receiving a request from the California Privacy Protection Agency, a deployer or a developer shall, within seven 30 days of the request, provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department. California Privacy Protection Agency.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)A trade secret, as defined in Section 3426.1 of the Civil Code, contained in an An impact assessment disclosed to the Civil Rights Department pursuant to this subdivision is California Privacy Protection Agency pursuant to this chapter, or that is subsequently shared by the California Privacy Protection Agency with another agency or department, shall be exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or a public prosecutor with whom an impact assessment was shared pursuant to subdivision (c), shall redact any trade secret, No provision of this chapter shall be construed to require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code, from the impact assessment. Code.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the Civil Rights Department. California Privacy Protection Agency.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
(c)The Civil Rights Department California Privacy Protection Agency may provide an impact assessment it receives to a public prosecutor listed in subdivision (a) of Section 22756.9 any of the following public entities to assist that public prosecutor entity in initiating or litigating a civil action under that section. action:
(d)The Civil Rights Department may adopt rules and regulations pursuant to the rulemaking provisions of the Administrative Procedure Act (Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code) as necessary to effectuate the intent of this section.(1)The Attorney General in the name of the people of the State of California.
(2)A district attorney, county counsel, or city attorney.
(3)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(4)The Civil Rights Department.
22756.9.(a)(1)Any of the following public entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D)The Civil Rights Department.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, or the Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.10.It shall be unlawful for a deployer, state government deployer, or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.11.This chapter does not apply to cybersecurity-related technology. technology, including technology designed to detect, protect against, or respond to security incidents, identity theft, fraud, harassment, malicious or deceptive activities or any illegal activity, preserve the integrity or security of systems, or investigate, report, or prosecute those responsible for those actions.
22756.12.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to the Civil Rights Department agencies and departments pursuant to Section 1 of this act remain confidential.
SEC. 3.The Legislature finds and declares that this act furthers the purposes and intent of the California Privacy Rights Act of 2020.
SEC. 3.SEC. 4.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means a system or service that uses artificial intelligence and has been specifically developed to, or specifically modified to, make, or be a substantial factor in making, consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions.
(F)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, local government agency, developer, or corporation that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)State government deployer means a state government agency that uses an automated decision tool to make a consequential decision. (h)(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(i)(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(j)(i)Significant update means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or outcome.
(j)State government deployer means a state government agency that uses an automated decision tool to make a consequential decision.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before first using it and annually thereafter.
(2)(A)With respect to an automated decision tool that a deployer first used before January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(B)This subdivision does not require a deployer to perform an impact assessment on an automated decision tool before using it if all of the following are true:
(i)The deployer uses the automated decision tool only for the intended use determined by the developer of the automated decision tool.
(ii)The deployer does not make any significant updates to the automated decision tool.
(iii)The developer of the automated decision tool has performed any impact assessment on the automated decision tool required by subdivision (c).
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment.
(7)A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity or relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
(e)A deployer or developer shall, in addition to the impact assessment required by subdivisions (a) and (b), shall perform, as soon as feasible, an impact assessment with respect to any significant update.
(f)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.2.(a)(1)A deployer shall, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
22756.3.(a)A developer shall provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following:
(1)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(2)A description of the type of data used to program or train the automated decision tool.
(3)A description of how the automated decision tool was evaluated for validity and explainability before sale or licensing.
(4)A description of the deployers responsibilities under this chapter.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.5.A deployer or developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)A deployer shall not use an automated decision tool that results in algorithmic discrimination.
(b)A developer shall not make available to potential deployers an automated decision tool that results in algorithmic discrimination.
22756.7.(a)A state government deployer shall, by January 1, 2026, provide the Civil Rights Department a list of automated decision tools initially deployed prior to January 1, 2025, by the state government deployer. The list shall identify all of the following:
(1)Each automated decision tool deployed by the state government deployer.
(2)The role of each automated decision tool in making consequential decisions.
(3)The population affected by each automated decision tool.
(b)The Civil Rights Department shall, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer shall comply with Sections 22756.1, 22756.2, 22756.3, 22756.4, 22756.5, and 22756.6 for each automated decision tool deployed by the state government deployer.
(1)The schedule established by the department shall prioritize compliance for automated decision tools used by state government deployers with the highest risk for adverse impacts, including civil rights violations and other discriminatory outcomes.
(2)The schedule established by the department shall require full compliance by each state government deployer by January 1, 2031.
(3)A state government deployer that fails to comply with the requirements of this chapter as specified in the schedule established by the department may be subject to enforcement by the department, as authorized in Sections 22756.8 and 22756.9.
(c)The Civil Rights Department may adopt rules and regulations pursuant to the rulemaking provisions of the Administrative Procedure Act (Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code) as necessary to effectuate the intent of this section.
22756.8.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon the request of the Civil Rights Department, a deployer or a developer shall, within seven days of the request, provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)A trade secret, as defined in Section 3426.1 of the Civil Code, contained in an impact assessment disclosed to the Civil Rights Department pursuant to this subdivision is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or a public prosecutor with whom an impact assessment was shared pursuant to subdivision (c), shall redact any trade secret, as defined in Section 3426.1 of the Civil Code, from the impact assessment.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the Civil Rights Department.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
(c)The Civil Rights Department may provide an impact assessment it receives to a public prosecutor listed in subdivision (a) of Section 22756.9 to assist that public prosecutor in initiating or litigating a civil action under that section.
(d)The Civil Rights Department may adopt rules and regulations pursuant to the rulemaking provisions of the Administrative Procedure Act (Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code) as necessary to effectuate the intent of this section.
22756.9.(a)(1)Any of the following public entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D)The Civil Rights Department.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, or the Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.10.It shall be unlawful for a deployer, state government deployer, or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.11.This chapter does not apply to cybersecurity-related technology.
22756.12.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to the Civil Rights Department pursuant to Section 1 of this act remain confidential.
SEC. 3.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means a system or service that uses artificial intelligence and has been specifically developed to, or specifically modified to, make, or be a substantial factor in making, consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions.
(F)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, state or local government agency, developer, or corporation that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)State government deployer means a state government agency that uses an automated decision tool to make a consequential decision.
(g)(h)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)(i)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)(j)Significant update means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or outcome.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before first using it and annually thereafter.
(2)With respect to an automated decision tool that a deployer first used before January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment.
(7)A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity or relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
(e)A deployer or developer shall, in addition to the impact assessment required by subdivisions (a) and (b), perform, as soon as feasible, an impact assessment with respect to any significant update.
(f)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.2.(a)(1)A deployer shall, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
22756.3.(a)A developer shall provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following:
(1)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(2)A description of the type of data used to program or train the automated decision tool.
(3)A description of how the automated decision tool was evaluated for validity and explainability before sale or licensing.
(4)A description of the deployers responsibilities under this chapter.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.5.A deployer or developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)A deployer shall not use an automated decision tool that results in algorithmic discrimination.
(b)A developer shall not make available to potential deployers an automated decision tool that results in algorithmic discrimination.
22756.7.(a)A state government deployer shall, by January 1, 2026, provide the Civil Rights Department a list of automated decision tools initially deployed prior to January 1, 2025, by the state government deployer. The list shall identify all of the following:
(1)Each automated decision tool deployed by the state government deployer.
(2)The role of each automated decision tool in making consequential decisions.
(3)The population affected by each automated decision tool.
(b)The Civil Rights Department shall, by January 1, 2027, establish a staggered schedule that identifies when each state government deployer shall comply with Sections 22756.1, 22756.2, 22756.3, 22756.4, 22756.5, and 22756.6 for each automated decision tool deployed by the state government deployer.
(1)The schedule established by the department shall prioritize compliance for automated decision tools used by state government deployers with the highest risk for adverse impacts, including civil rights violations and other discriminatory outcomes.
(2)The schedule established by the department shall require full compliance by each state government deployer by January 1, 2031.
(3)A state government deployer that fails to comply with the requirements of this chapter as specified in the schedule established by the department may be subject to enforcement by the department, as authorized in Sections 22756.8 and 22756.9.
(c)The Civil Rights Department may adopt rules and regulations pursuant to the rulemaking provisions of the Administrative Procedure Act (Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code) as necessary to effectuate the intent of this section.
22756.7.22756.8.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon the request of the Civil Rights Department, a deployer or a developer shall, within seven days of the request, provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)A trade secret, as defined in Section 3426.1 of the Civil Code, contained in an impact assessment disclosed to the Civil Rights Department pursuant to this subdivision is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or a public prosecutor with whom an impact assessment was shared pursuant to subdivision (c), shall redact any trade secret, as defined in Section 3426.1 of the Civil Code, from the impact assessment.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the Civil Rights Department.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
(c)The Civil Rights Department may provide an impact assessment it receives to a public prosecutor listed in subdivision (a) of Section 22756.8 22756.9 to assist that public prosecutor in initiating or litigating a civil action under that section.
(d)The Civil Rights Department may adopt rules and regulations pursuant to the rulemaking provisions of the Administrative Procedure Act (Chapter 3.5 (commencing with Section 11340) of Part 1 of Division 3 of Title 2 of the Government Code) as necessary to effectuate the intent of this section.
22756.8.22756.9.(a)(1)Any of the following public entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D)The Civil Rights Department.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, or the Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.9.22756.10.It shall be unlawful for a deployer deployer, state government deployer, or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.10.22756.11.This chapter does not apply to cybersecurity-related technology.
22756.11.22756.12.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws laws, including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to the Civil Rights Department pursuant to Section 1 of this act remain confidential.
SEC. 3.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
(b)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means a system or service that uses artificial intelligence and has been specifically developed to, or specifically modified to, make, or be a substantial factor in making, consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions.
(F)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, state or local government agency, developer, or corporation that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)Significant update means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or outcome.
22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before first using it and annually thereafter.
(2)With respect to an automated decision tool that a deployer first used before January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment.
(7)A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity or relevance.
(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
(3)A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
(e)A deployer or developer shall, in addition to the impact assessment required by subdivisions (a) and (b), perform, as soon as feasible, an impact assessment with respect to any significant update.
(f)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.2.(a)(1)A deployer shall, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
22756.3.(a)A developer shall provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following:
(1)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(2)A description of the type of data used to program or train the automated decision tool.
(3)A description of how the automated decision tool was evaluated for validity and explainability before sale or licensing.
(4)A description of the deployers responsibilities under this chapter.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.5.A deployer or developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)A deployer shall not use an automated decision tool that results in algorithmic discrimination.
(b)A developer shall not make available to potential deployers an automated decision tool that results in algorithmic discrimination.
22756.7.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon the request of the Civil Rights Department, a deployer or a developer shall, within seven days of the request, provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)A trade secret, as defined in Section 3426.1 of the Civil Code, contained in an impact assessment disclosed to the Civil Rights Department pursuant to this subdivision is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or an entity a public prosecutor with which whom an impact assessment was shared pursuant to subdivision (d), (c), shall redact any trade secret, as defined in Section 3426.1 of the Civil Code, from the impact assessment.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the Civil Rights Department.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
(c)The Civil Rights Department may provide an impact assessment it receives to a public prosecutor listed in subdivision (a) of Section 22756.8 to assist that public prosecutor in initiating or litigating a civil action under that section.
(d)The Civil Rights Department may adopt rules and regulations as necessary to effectuate the intent of this section.
22756.8.(a)(1)Any of the following public entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D)The Civil Rights Department.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, attorney, or the Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.9.It shall be unlawful for a deployer or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.10.This chapter does not apply to cybersecurity-related technology.
22756.11.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to the Civil Rights Department pursuant to Section 1 of this act remain confidential.
SEC. 3.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
(b)Artificial intelligence means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing a real or virtual environment. an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(c)Automated decision tool means a system or service that uses artificial intelligence and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling substantial factor in making, consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions.
(F)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, state or local government agency, developer, or corporation that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a controlling substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)Significant update means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or expected outcome.
22756.1.(a)On or before January 1, 2026, and annually thereafter, a deployer of an automated decision tool shall perform an impact assessment for any automated decision tool the deployer uses that includes all of the following:22756.1.(a)(1)Subject to paragraph (2), a deployer shall perform an impact assessment on any automated decision tool before first using it and annually thereafter.
(2)With respect to an automated decision tool that a deployer first used before January 1, 2025, the deployer shall perform an impact assessment on that automated decision tool before January 1, 2026, and annually thereafter.
(b)A deployer shall ensure that an impact assessment prepared pursuant to subdivision (a) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a controlling substantial factor in making, a consequential decision.
(3)A summary of the type of data categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a controlling substantial factor in making, a consequential decision. decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment.
(7)A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a controlling substantial factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity or relevance.
(b)On or before January 1, 2026, and annually thereafter, a developer of an automated decision tool shall complete and document an impact assessment of any automated decision tool that it designs, codes, or produces that includes all of the following:(c)(1)Subject to paragraph (2), a developer, before making an automated decision tool that it designs, codes, or produces available to potential deployers, shall perform an impact assessment on the automated decision tool and annually thereafter.
(2)With respect to an automated decision tool that a developer first made available to potential deployers before January 1, 2025, the developer shall perform an impact assessment on the automated decision tool before January 1, 2026, and annually thereafter.
(d)A developer shall ensure that an impact assessment prepared pursuant to subdivision (c) includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a controlling substantial factor in making, a consequential decision.
(3)A summary of the type of data categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a controlling substantial factor in making, a consequential decision. decision, including, but not limited to, all of the following:
(A)Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B)Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C)Each category of information related to a natural persons receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
(4)An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or monitored when it is used, to make, or be a controlling substantial factor in making, a consequential decision.
(c)(e)A deployer or developer shall, in addition to the impact assessment required by subdivisions (a) and (b), perform, as soon as feasible, an impact assessment with respect to any significant update.
(d)(f)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.2.(a)(1)A deployer shall, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision. used.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision.
(D)Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
22756.3.(a)A developer shall provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following:
(1)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(2)A description of the type of data used to program or train the automated decision tool.
(3)A description of how the automated decision tool was evaluated for validity and explainability before sale or licensing.
(4)A description of the deployers responsibilities under this chapter.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for two five years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.5.A deployer or developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.(a)A deployer shall not use an automated decision tool that results in algorithmic discrimination.
(b)A developer shall not make available to potential deployers an automated decision tool that results in algorithmic discrimination.
22756.7.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon the request of the Civil Rights Department, a deployer or a developer shall, within seven days of the request, provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)A trade secret, as defined in Section 3426.1 of the Civil Code, contained in an impact assessment disclosed to the Civil Rights Department pursuant to this subdivision is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or an entity with which an impact assessment was shared pursuant to subdivision (d), shall redact any trade secret, as defined in Section 3426.1 of the Civil Code, from the impact assessment.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the Civil Rights Department.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
(c)The Civil Rights Department may provide an impact assessment it receives to a public prosecutor listed in subdivision (a) of Section 22756.8 to assist that public prosecutor in initiating or litigating a civil action under that section.
22756.8.(a)(1)Any of the following public attorneys entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D)The Civil Rights Department.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.9.It shall be unlawful for a deployer or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.10.This chapter does not apply to cybersecurity-related technology.
22756.11.The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to the Civil Rights Department pursuant to Section 1 of this act remain confidential.
SEC. 3.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
As used in this chapter:
(a)Algorithmic discrimination means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
(b)Artificial intelligence means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing a real or virtual environment.
(c)Automated decision tool means a system or service that uses artificial intelligence and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling factor in making, consequential decisions.
(d)Consequential decision means a decision or judgment that has a legal, material, or similarly significant effect on an individuals life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of, any of the following:
(1)Employment with respect to all of the following:
(A)Pay or promotion.
(B)Hiring or termination.
(C)Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.
(2)Education and vocational training as it relates to all of the following:
(A)Assessment.
(B)Detecting student cheating or plagiarism.
(C)Accreditation.
(D)Certification.
(E)Admissions.
(F)Financial aid or scholarships.
(3)Housing or lodging, including rental or short-term housing or lodging.
(4)All of the following essential utilities:
(A)Electricity.
(B)Heat.
(C)Water.
(D)Internet or telecommunications access.
(E)Transportation.
(5)Family planning.
(6)Adoption services, reproductive services, or assessments related to child protective services.
(7)Health care or health insurance, including mental health care, dental, or vision.
(8)Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor.
(9)All of the following aspects of the criminal justice system:
(A)Risk assessments for pretrial hearings.
(B)Sentencing.
(C)Parole.
(10)Legal services.
(11)Private arbitration.
(12)Mediation.
(13)Voting.
(e)Deployer means a person, partnership, state or local government agency, or corporation that uses an automated decision tool to make a consequential decision.
(f)Developer means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a controlling factor in making, consequential decisions, whether for its own use or for use by a third party.
(g)Impact assessment means a documented risk-based evaluation of an automated decision tool that meets the criteria of Section 22756.1.
(h)Sex includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation.
(i)Significant update means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or expected outcome.
22756.1.(a)On or before January 1, 2026, and annually thereafter, a deployer of an automated decision tool shall perform an impact assessment for any automated decision tool the deployer uses that includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a controlling factor in making, a consequential decision.
(3)A summary of the type of data collected from natural persons and processed by the automated decision tool when it is used to make, or be a controlling factor in making, a consequential decision.
(4)A statement of the extent to which the deployers use of the automated decision tool is consistent with or varies from the statement required of the developer by Section 22756.3.
(5)An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(6)A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment.
(7)A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a controlling factor in making, a consequential decision.
(8)A description of how the automated decision tool has been or will be evaluated for validity or relevance.
(b)On or before January 1, 2026, and annually thereafter, a developer of an automated decision tool shall complete and document an impact assessment of any automated decision tool that it designs, codes, or produces that includes all of the following:
(1)A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts.
(2)A description of the automated decision tools outputs and how they are used to make, or be a controlling factor in making, a consequential decision.
(3)A summary of the type of data collected from natural persons and processed by the automated decision tool when it is used to make, or be a controlling factor in making, a consequential decision.
(4)An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployers use of the automated decision tool.
(5)A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the automated decision tool.
(6)A description of how the automated decision tool can be used by a natural person, or monitored when it is used, to make, or be a controlling factor in making, a consequential decision.
(c)A deployer or developer shall, in addition to the impact assessment required by subdivisions (a) and (b), perform, as soon as feasible, an impact assessment with respect to any significant update.
(d)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.2.(a)(1)A deployer shall, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision.
(2)A deployer shall provide to a natural person notified pursuant to this subdivision all of the following:
(A)A statement of the purpose of the automated decision tool.
(B)Contact information for the deployer.
(C)A plain language description of the automated decision tool that includes a description of any human components and how any automated component is used to inform a consequential decision.
(b)(1)If a consequential decision is made solely based on the output of an automated decision tool, a deployer shall, if technically feasible, accommodate a natural persons request to not be subject to the automated decision tool and to be subject to an alternative selection process or accommodation.
(2)After a request pursuant to paragraph (1), a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer shall not be obligated to provide an alternative selection process or accommodation.
22756.3.(a)A developer shall provide a deployer with a statement regarding the intended uses of the automated decision tool and documentation regarding all of the following:
(1)The known limitations of the automated decision tool, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
(2)A description of the type of data used to program or train the automated decision tool.
(3)A description of how the automated decision tool was evaluated for validity and explainability before sale or licensing.
(b)This section does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
22756.4.(a)(1)A deployer or developer shall establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an automated decision tool.
(2)The safeguards required by this subdivision shall be appropriate to all of the following:
(A)The use or intended use of the automated decision tool.
(B)The deployers or developers role as a deployer or developer.
(C)The size, complexity, and resources of the deployer or developer.
(D)The nature, context, and scope of the activities of the deployer or developer in connection with the automated decision tool.
(E)The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an automated decision tool.
(b)The governance program required by this section shall be designed to do all of the following:
(1)(A)Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this chapter.
(B)(i)An employee designated pursuant to this paragraph shall have the authority to assert to the employees employer a good faith belief that the design, production, or use of an automated decision tool fails to comply with the requirements of this chapter.
(ii)An employer of an employee designated pursuant to this paragraph shall conduct a prompt and complete assessment of any compliance issue raised by that employee.
(2)Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an automated decision tool.
(3)If established by a deployer, provide for the performance of impact assessments as required by Section 22756.1.
(4)If established by a developer, provide for compliance with Sections 22756.2 and 22756.3.
(5)Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this chapter.
(6)Maintain for two years after completion the results of an impact assessment.
(7)Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the automated decision tool, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
(c)This section does not apply to a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an automated decision tool that impacted more than 999 people per year.
22756.5.A deployer or developer shall make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
(a)The types of automated decision tools currently in use or made available to others by the deployer or developer.
(b)How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the automated decision tools it currently uses or makes available to others.
22756.6.A deployer shall not use an automated decision tool that results in algorithmic discrimination.
22756.7.(a)The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
(b)(1)Upon the request of the Civil Rights Department, a deployer or a developer shall, within seven days of the request, provide any impact assessment that it performed pursuant to this chapter to the Civil Rights Department.
(2)The disclosure of an impact assessment pursuant to this subdivision does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
(3)(A)A trade secret, as defined in Section 3426.1 of the Civil Code, contained in an impact assessment disclosed to the Civil Rights Department pursuant to this subdivision is exempt from the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code).
(B)In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or an entity with which an impact assessment was shared pursuant to subdivision (d), shall redact any trade secret, as defined in Section 3426.1 of the Civil Code, from the impact assessment.
(4)A deployer or developer who violates this subdivision shall be liable for an administrative fine of not more than ten thousand dollars ($10,000) per violation in an administrative enforcement action brought by the Civil Rights Department.
(5)Each day on which an automated decision tool is used for which an impact assessment has not been submitted pursuant to this subdivision shall give rise to a distinct violation of this subdivision.
22756.8.(a)(1)Any of the following public attorneys may bring a civil action against a deployer or developer for a violation of this chapter:
(A)The Attorney General in the name of the people of the State of California.
(B)A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C)A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(2)A court may award in an action brought pursuant to this subdivision all of the following:
(A)Injunctive relief.
(B)Declaratory relief.
(C)Reasonable attorneys fees and litigation costs.
(D)Only in an action for a violation involving algorithmic discrimination, a civil penalty of twenty-five thousand dollars ($25,000) per violation.
(b)(1)A public attorney, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days written notice to a deployer or developer of the alleged violations of this chapter.
(2)(A)The developer or deployer may cure, within 45 days of receiving the written notice described in paragraph (1), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured.
(B)If the developer or deployer cures the noticed violation and provides the express written statement pursuant to subparagraph (A), a claim for injunctive relief shall not be maintained for the noticed violation.
22756.9.It shall be unlawful for a deployer or developer to retaliate against a natural person for that persons exercise of rights provided for under this chapter.
22756.10.This chapter does not apply to cybersecurity-related technology.
SEC. 2.The Legislature finds and declares that Section 1 of this act, which adds Chapter 25 (commencing with Section 22756) to Division 8 of the Business and Professions Code, imposes a limitation on the publics right of access to the meetings of public bodies or the writings of public officials and agencies within the meaning of Section 3 of Article I of the California Constitution. Pursuant to that constitutional provision, the Legislature makes the following findings to demonstrate the interest protected by this limitation and the need for protecting that interest:
In order to protect proprietary information, it is necessary that trade secrets disclosed in impact assessments to the Civil Rights Department pursuant to Section 1 of this act remain confidential.
SEC. 3.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
|
Date
|
Status
|
|---|---|
| Aug 31, 2024 | Ordered to inactive file at the request of Senator Umberg. |
| Aug 29, 2024 | Read second time. Ordered to third reading. |
| Aug 28, 2024 | Read third time and amended. Ordered to second reading. |
| Aug 28, 2024 | Joint Rule 61 suspended. (Ayes 31. Noes 9.) |
| Aug 19, 2024 | Read second time. Ordered to third reading. |
| Aug 15, 2024 | Read second time and amended. Ordered returned to second reading. |
| Aug 15, 2024 | From committee: Amend, and do pass as amended. (Ayes 4. Noes 2.) (August 15). |
| Aug 5, 2024 | In committee: Referred to APPR suspense file. |
| Jul 3, 2024 | Read second time and amended. Re-referred to Com. on APPR. |
| Jul 3, 2024 | From committee: Amend, and do pass as amended and re-refer to Com. on APPR. (Ayes 9. Noes 2.) (July 2). |
| Jun 24, 2024 | From committee chair, with author's amendments: Amend, and re-refer to committee. Read second time, amended, and re-referred to Com. on JUD. |
| Jun 3, 2024 | From committee chair, with author's amendments: Amend, and re-refer to committee. Read second time, amended, and re-referred to Com. on JUD. |
| May 29, 2024 | Referred to Com. on JUD. |
| May 22, 2024 | In Senate. Read first time. To Com. on RLS. for assignment. |
| May 21, 2024 | Read third time. Passed. Ordered to the Senate. (Ayes 50. Noes 14. Page 5394.) |
| May 20, 2024 | Read second time. Ordered to third reading. |
| May 16, 2024 | Read second time and amended. Ordered returned to second reading. |
| May 16, 2024 | From committee: Amend, and do pass as amended. (Ayes 11. Noes 4.) (May 16). |
| May 16, 2024 | Assembly Rule 63 suspended. |
| May 8, 2024 | In committee: Set, first hearing. Referred to APPR. suspense file. |
| Apr 25, 2024 | Re-referred to Com. on APPR. |
| Apr 24, 2024 | Read second time and amended. |
| Apr 23, 2024 | Re-referred to Com. on JUD. |
| Apr 23, 2024 | From committee: Amend, and do pass as amended and re-refer to Com. on APPR. (Ayes 9. Noes 2.) (April 23). |
| Apr 22, 2024 | Read second time and amended. |
| Apr 18, 2024 | From committee: Amend, and do pass as amended and re-refer to Com. on JUD. (Ayes 8. Noes 3.) (April 16). |
| Apr 18, 2024 | Assembly Rule 56 suspended. |
| Apr 18, 2024 | (Pending re-refer to Com. on JUD.) |
| Mar 21, 2024 | Referred to Coms. on P. & C.P. and JUD. |
| Feb 16, 2024 | From printer. May be heard in committee March 17. |
| Feb 15, 2024 | Read first time. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Aug 15, 2024 | Do pass as amended | Senate Standing Committee on Appropriations |
Passed |
|
| Aug 5, 2024 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Jul 2, 2024 | Do pass as amended, but first amend, and re-refer to the Committee on [Appropriations] | Senate Standing Committee on Judiciary |
Passed |
|
| May 21, 2024 | AB 2930 Bauer-Kahan Assembly Third Reading | Assembly Floor |
Passed |
|
| May 16, 2024 | Do pass as amended. | Assembly Standing Committee on Appropriations |
Passed |
|
| Apr 23, 2024 | Do pass as amended and be re-referred to the Committee on [Appropriations] | Assembly Standing Committee on Judiciary |
Passed |
|
| Apr 16, 2024 | Do pass as amended and be re-referred to the Committee on [Judiciary] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| Aclu California Action | ** Against | |
| Advanced Medical Technology Association | ** Against | |
| Alliance for Digital Innovation | ** Against | |
| American Council Of Life Insurers | ** Against | |
| American Federation of Musicians, Local 47 | ** Support | |
| American Property Casualty Insurance Association | ** Against | |
| American Staffing Association | ** Against | |
| AmericaS Physician Groups | ** Against | |
| Association Of California Life & Health Insurance Companies | ** Against | |
| Association of National Advertisers | ** Against | |
| Bsa The Software Alliance | ** Support | |
| CalBroadband | ** Against | |
| California Association Of Health Plans | ** Against | |
| California Association Of Realtors | * Against | Apr 23, 2024 |
| California Bankers Association | ** Against | |
| California Chamber Of Commerce | * Against | Jul 2, 2024 |
| California Communications Association | ** Against | |
| California Community Banking Network | ** Against | |
| California Credit Union League | ** Against | |
| California Employment Lawyers Association | ** Support | |
| California Financial Services Association | ** Against | |
| California Hospital Association | ** Against | |
| California Hotel & Lodging Association | ** Against | |
| California Life Sciences | ** Against | |
| California Manufacturers & Technology Association | * Against | Jul 2, 2024 |
| California Medical Association | ** Against | |
| California Mortgage Bankers Association | ** Against | |
| California Nurses Association | ** Support | |
| California Radiological Society | ** Against | |
| California Retailers Association | ** Against | |
| California Staffing and Recruiting Association | ** Against | |
| California Staffing Professionals | ** Against | |
| Capitol Advocacy | * Against | Apr 16, 2024 |
| Capitol Advocacy | * Against Unless Amend | Apr 16, 2024 |
| Center for Democracy and Technology | ** Support | |
| Center on Race and Digital Justice | ** Support | |
| Center on Race and Digital Justice Secure Justice | ** Support | |
| Chamber Of Progress | ** Against | |
| Civil Justice Association Of California | * Against Unless Amend | Jul 2, 2024 |
| Computer And Communications Industry Association | ** Against | |
| Consumer Reports | ** Support | |
| Consumer Technology Association | ** Against | |
| Ctia The Wireless Association | ** Against | |
| East Bay Community Law Center | ** Support | |
| Economic Security California Action | ** Support | |
| Electronic Frontier Foundation | ** Against | |
| Equal Rights Advocates | ** Support | |
| ** Against | ||
| Insights Association | ** Against | |
| Internet.Works | ** Against | |
| Kaiser Permanente | ** Against | |
| Legal Aid At Work | ** Support | |
| Los Angeles Area Chamber Of Commerce | ** Against | |
| Microsoft Corporation | ** Support | |
| Mortgage Bankers Association | ** Against | |
| National Association of Mutual Insurance Companies | ** Against | |
| Oakland Privacy | ** Support | |
| Orange County Business Council | ** Against | |
| Pacific Association Of Domestic Insurance Companies | ** Against | |
| Personal Insurance Federation Of California | ** Against | |
| Rise Economy | ** Support | |
| Secure Justice | ** Support | |
| SEIU California | ** Support | |
| State Privacy and Security Coalition, Inc. | ** Against | |
| Sutter Health | ** Against | |
| TechEquity Collaborative | ** Support | |
| Technet | * Against | Jul 2, 2024 |
| The Greenlining Institute | ** Support | |
| UC Berkeley | * Support | Apr 23, 2024 |
| Ustelecom | ** Against | |
| VERIZON COMMUNICATIONS | ** Against | |
| Workday | ** Support | |
| Workday Inc | * Support | Apr 16, 2024 |
| Zella Life | * Against | Jul 2, 2024 |
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | AB 2930 |
THIRD READING
Bill No: AB 2930
Author: Bauer-Kahan (D)
Amended: 8/28/24 in Senate
Vote: 21
SENATE JUDICIARY COMMITTEE: 9-2, 7/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Roth, Stern, Wahab
NOES: Wilk, Niello
SENATE APPROPRIATIONS COMMITTEE: 4-2, 8/15/24
AYES: Caballero, Ashby, Becker, Wahab
NOES: Jones, Seyarto
NO VOTE RECORDED: Bradford
ASSEMBLY FLOOR: 50-14, 5/21/24 - See last page for vote
SUBJECT: Automated decision systems
SOURCE: Author
DIGEST:
This bill regulates the use of “automated decision systems” (ADS) in order to prevent “algorithmic discrimination.” This includes requirements on developers and deployers that make and use these tools to make “consequential decisions” to perform impact assessments on ADSs. This bill establishes the right of individuals to know when an ADS is being used, the right to opt out of its use, and an explanation of how it is used.
Senate Floor amendments of 8/28/24 update definition for automated decision system, ensure proper oversight by the Civil Rights Department, and clarify its interaction with other laws.
ANALYSIS:
Existing law:
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act. (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
This bill:
- Requires a deployer to perform an impact assessment on any ADS before the system is first deployed and annually thereafter. With respect to an ADS that a deployer first used prior to January 1, 2025, the deployer shall perform an impact assessment on that ADS before January 1, 2026, and annually thereafter.
- Provides, notwithstanding the above, that a deployer is not required to perform an impact assessment on an ADS before using it if specified conditions are met.
- Requires a deployer to ensure that the above impact assessment includes all of the following:
- A statement of the purpose of the ADS and its intended benefits, uses, and deployment contexts.
- A description of specified features of the ADS, including the personal characteristics or attributes that the ADS will measure or assess, the method for doing so, and how they are relevant to the consequential decisions for which the ADS will be used, as well as information on its outputs.
- A summary of the categories of information collected from natural persons and processed by the ADS when it is used to make, or be a substantial factor in making, a consequential decision, including categories of sensitive information and information related to a natural person’s receipt of sensitive services.
- A statement of the extent to which the deployer’s use of the ADS is consistent with or varies from the statement required of the developer.
- An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of specified protected categories, resulting from the deployer’s use of the ADS.
- A description of the safeguards implemented, or that will be implemented, to address reasonably foreseeable risks of algorithmic discrimination that address specified matters.
- A description of how the ADS will be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
- A description of how the ADS has been or will be evaluated for validity, reliability, and relevance.
- Requires a developer to perform an impact assessment on an ADS and annually thereafter, as provided, which must include specified information.
- Requires a deployer or developer to perform, as soon as feasible, an impact assessment with respect to a substantial modification to an ADS.
- Requires a deployer, prior to an ADS making a consequential decision, or being a substantial factor in making a consequential decision, to notify any natural person that is subject to the consequential decision that an ADS is being used and to provide specified information.
- Requires a deployer, if a consequential decision is made solely based on the output of an ADS, to, if technically feasible, accommodate a natural person’s request to not be subject to the ADS and to instead be subject to an alternative selection process or accommodation, as specified.
- Requires a deployer that has deployed an ADS, to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide that person with specified information about the ADS.
- Requires the notices and other communications described above to meet specified conditions, including that they be in clear and plain language.
- Requires a developer to provide a deployer with the results of any impact assessment performed on an ADS, as provided, along with specified documentation.
- States that the above does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
- Requires a deployer or developer to establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards, as specified. The program shall provide for annual and comprehensive reviews of policies, practices, and procedures to ensure compliance.
- Requires a deployer and developer to make publicly available a clear policy that provides a summary of the types of ADSs currently in use or made available to others and how they manage the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the ADSs.
- Provides that if an impact assessment performed by a deployer identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the ADS until the risk has been mitigated. If an impact assessment performed by a developer identifies such a risk under deployment conditions reasonably likely to occur in this state, the developer shall not make the ADS available to potential deployers until the risk has been mitigated.
- Authorizes the Civil Rights Department (CRD) to investigate violations and to request impact assessments from deployers and developers.
- to bring a civil action against a deployer or developer in violation. Provides that, in such an action, a court may award injunctive and declaratory relief, as well as attorneys’ fees and costs. In an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation may also be awarded. Defendants in such cases are provided a 45-day right to cure.
- Makes it unlawful for a deployer or developer to retaliate against a natural person for that person’s exercise of rights provided herein.
- Provides certain exemptions and clarifies that the rights, remedies, and penalties established therein are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws.
- Defines the relevant terms, including:
- “Algorithmic discrimination” means the condition in which an ADS contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
- “Automated decision system” means, consistent with Section 11546.45.5 of the Government Code, a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and that is used to make, or be a substantial factor in making, a consequential decision.
- “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of employment, as provided.
- “Deployer” means a person, partnership, developer, corporation, or any contractor or agent of those entities, that uses an ADS to make a consequential decision.
- “Developer” means a person, partnership, or corporation that designs, codes, or produces an ADS, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
- “Substantial factor” means an element of a decisionmaking process that is capable of altering the outcome of the process.
- “Substantial modification” means a new version, new release, or other update to an ADS that materially changes its uses, intended uses, or outcomes.
Background
Automated decisionmaking is one particular area where AI is being increasingly deployed. Major transparency and fairness concerns have been raised about the use of ADSs to make consequential decisions, essentially determinations with significant legal or other material effect on one’s life. This bill seeks to regulate their use by requiring impact assessments to evaluate their purpose, use of data, potential for bias, and the steps taken to address those risks. The bill also ensures that individuals that are subject to ADSs know when the tool is being used to make a “consequential decision” about them, are able to opt out of their use, and are given a reasonable explanation for the automated decision made and a chance to correct any incorrect data.
This bill is author-sponsored. It is supported by various organizations, including TechEquity Action and Legal Aid at Work. It is opposed by various industry associations, including Google and the American Council of Life Insurers. For a more thorough assessment, please see the Senate Judiciary Committee analysis.
Comments
According to the author:
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADSs) that make consequential decisions and mitigate any discovered biases. The use of ADS’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADSs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes
According to the Senate Appropriations Committee: Unknown, potentially significant ongoing costs (General Fund) to the Civil Rights Division (CRD). The number of ADSs that this bill would regulate is unknown, but the number is surely very large and growing. Therefore, an unknown but significant number of complaints may be filed with the department under this bill, which CRD may prosecute. These prosecutions will involve advanced technologies and complex legal analyses and require highly-skilled investigators and attorneys working with technology and statistical experts. Should this bill be enacted, CRD would likely need a significant number of permanent positions and other resources, depending on the number and complexity of complaints filed with CRD. This would include funding for permanent positions across all of CRD’s divisions as well as funding to hire expert consultants.
SUPPORT: (Verified 8/28/24)
American Federation of Musicians, Local 7
California Employment Lawyers Association
Center for Democracy and Technology
Center on Race and Digital Justice Secure Justice
Consumer Reports
East Bay Community Law Center
Economic Security California Action
Equal Rights Advocates
The Greenlining Institute
Legal Aid at Work
Rise Economy
Techequity Collaborative
OPPOSITION: (Verified 8/28/24)
ACLU California Action
Advanced Medical Technology Association
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
America’s Physician Groups
Association of California Life & Health Insurance Companies
CalBroadband
California Association of Health Plans
California Bankers Association
California Communications Association
California Community Banking Network
California Financial Services Association
California Hospital Association
California Life Sciences
California Medical Association
California Mortgage Bankers Association
California Staffing Professionals
Consumer Technology Association
CTIA – The Wireless Association
Electronic Frontier Foundation
Kaiser Permanente
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Orange County Business Council
Pacific Association of Domestic Insurance Companies
Personal Insurance Federation of California
Sutter Health
USTelecom
Verizon Communications
ARGUMENTS IN SUPPORT: A coalition of groups, including Consumer Reports, Equal Rights Advocates, and the Greenlining Institute, write
AB 2930 would enact common-sense guardrails to help ensure that developers and deployers of these tools are obligated to test and mitigate for discriminatory outcomes prior to the sale or use of these tools in our communities. Specifically, the legislation would:
1. Require developers and deployers to conduct pre-deployment impact assessments to determine any potential for discrimination on people with protected class status;
2. Prohibit the sale or use of an ADS that may create a discriminatory outcome on people within a protected class until that adverse impact has been addressed and resolved;
3. Provide consumers with pre-use notice of the tool, a post-use explanation, the right to correct inaccurate information, and access to alternative selection procedures.
ARGUMENTS IN OPPOSITION: The Consumer Technology Association writes:
AB 2930 would require impact assessments be performed for “any” ADS the deployer uses, regardless of whether the use of the tool presents any significant risks. A risk-based approach to regulating AI tools is necessary to avoid overbroad regulations and costly new mandates that can, and should, be narrowly focused on only those use cases presenting greatest risks to individuals or society.
ASSEMBLY FLOOR: 50-14, 5/21/24
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Wendy Carrillo, Connolly, Mike Fong, Garcia, Gipson, Grayson, Haney, Hart, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Ortega, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Rendon, Reyes, Santiago, Schiavo, Ting, Villapudua, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas
NOES: Alanis, Chen, Davies, Dixon, Flora, Vince Fong, Gallagher, Hoover, Lackey, Jim Patterson, Joe Patterson, Sanchez, Ta, Waldron
NO VOTE RECORDED: Bains, Juan Carrillo, Cervantes, Megan Dahle, Essayli, Friedman, Gabriel, Holden, Mathis, Ramos, Luz Rivas, Rodriguez, Blanca Rubio, Soria, Valencia, Wallis
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
8/29/24 16:35:31
**** END ****
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | AB 2930 |
THIRD READING
Bill No: AB 2930
Author: Bauer-Kahan (D)
Amended: 8/15/24 in Senate
Vote: 21
SENATE JUDICIARY COMMITTEE: 9-2, 7/2/24
AYES: Umberg, Allen, Ashby, Caballero, Durazo, Laird, Roth, Stern, Wahab
NOES: Wilk, Niello
SENATE APPROPRIATIONS COMMITTEE: 4-2, 8/15/24
AYES: Caballero, Ashby, Becker, Wahab
NOES: Jones, Seyarto
NO VOTE RECORDED: Bradford
ASSEMBLY FLOOR: 50-14, 5/21/24 - See last page for vote
SUBJECT: Automated decision tools
SOURCE: Author
DIGEST:
This bill regulates the use of “automated decision tools” (ADTs) in order to prevent “algorithmic discrimination.” This includes requirements on developers and deployers that make and use these tools to make “consequential decisions” to perform impact assessments on ADTs. The bill establishes the right of individuals to know when an ADT is being used, the right to opt out of its use, and an explanation of how it is used.
ANALYSIS:
Existing law:
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act. (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
This bill:
- Requires a deployer to perform an impact assessment on any ADT before the tool is first deployed and annually thereafter. With respect to an ADT that a deployer first used prior to January 1, 2025, the deployer shall perform an impact assessment on that ADT before January 1, 2026, and annually thereafter.
- Provides, notwithstanding the above, that a deployer is not required to perform an impact assessment on an ADT before using it if specified conditions are met.
- Requires a deployer to ensure that the above impact assessment includes all of the following:
- A statement of the purpose of the ADT and its intended benefits, uses, and deployment contexts.
- A description of specified features of the ADT, including the personal characteristics or attributes that the ADT will measure or assess, the method for doing so, and how they are relevant to the consequential decisions for which the ADT will be used, as well as information on its outputs.
- A summary of the categories of information collected from natural persons and processed by the ADT when it is used to make, or be a substantial factor in making, a consequential decision, including categories of sensitive information and information related to a natural person’s receipt of sensitive services.
- A statement of the extent to which the deployer’s use of the ADT is consistent with or varies from the statement required of the developer.
- An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of specified protected categories, resulting from the deployer’s use of the ADT.
- A description of the safeguards implemented, or that will be implemented, to address reasonably foreseeable risks of algorithmic discrimination that address specified matters.
- A description of how the ADT will be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
- A description of how the ADT has been or will be evaluated for validity, reliability, and relevance.
- Requires a developer to perform an impact assessment on an ADT and annually thereafter, as provided, which must include specified information.
- Requires a deployer or developer to perform, as soon as feasible, an impact assessment with respect to a substantial modification to an ADT.
- Requires a deployer, prior to an ADT making a consequential decision, or being a substantial factor in making a consequential decision, to notify any natural person that is subject to the consequential decision that an ADT is being used and to provide specified information.
- Requires a deployer, if a consequential decision is made solely based on the output of an ADT, to, if technically feasible, accommodate a natural person’s request to not be subject to the ADT and to instead be subject to an alternative selection process or accommodation, as specified.
- Requires a deployer that has deployed an ADT, to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide that person with specified information about the ADT.
- Requires the notices and other communications described above to meet specified conditions, including that they be in clear and plain language.
- Requires a developer to provide a deployer with the results of any impact assessment performed on an ADT, as provided, along with specified documentation.
- States that the above does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
- Requires a deployer or developer to establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards, as specified. The program shall provide for annual and comprehensive reviews of policies, practices, and procedures to ensure compliance.
- Requires a deployer and developer to make publicly available a clear policy that provides a summary of the types of ADTs currently in use or made available to others and how they manage the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the ADTs.
- Provides that if an impact assessment performed by a deployer identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the ADT until the risk has been mitigated. If an impact assessment performed by a developer identifies such a risk under deployment conditions reasonably likely to occur in this state, the developer shall not make the ADT available to potential deployers until the risk has been mitigated.
- Authorizes the Civil Rights Department (CRD) to bring a civil action against a deployer or developer in violation. Provides that, in such an action, a court may award injunctive and declaratory relief, as well as attorneys’ fees and costs. In an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation may also be awarded. Defendants in such cases are provided a 45-day right to cure.
- Makes it unlawful for a deployer or developer to retaliate against a natural person for that person’s exercise of rights provided herein.
- Provides certain exemptions and clarifies that the rights, remedies, and penalties established therein are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws.
- Defines the relevant terms, including:
- “Algorithmic discrimination” means the condition in which an ADT contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
- “Automated decision tool” means an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
- “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of employment, as provided.
- “Deployer” means a person, partnership, developer, corporation, or any contractor or agent of those entities, that uses an ADT to make a consequential decision.
- “Developer” means a person, partnership, or corporation that designs, codes, or produces an ADT, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
- “Substantial factor” means an element of a decisionmaking process that is capable of altering the outcome of the process.
- “Substantial modification” means a new version, new release, or other update to an ADT that materially changes its uses, intended uses, or outcomes.
Background
Automated decisionmaking is one particular area where AI is being increasingly deployed. Major transparency and fairness concerns have been raised about the use of ADTs to make consequential decisions, essentially determinations with significant legal or other material effect on one’s life. This bill seeks to regulate their use by requiring impact assessments to evaluate their purpose, use of data, potential for bias, and the steps taken to address those risks. The bill also ensures that individuals that are subject to ADTs know when the tool is being used to make a “consequential decision” about them, are able to opt out of their use, and are given a reasonable explanation for the automated decision made and a chance to correct any incorrect data.
This bill is author-sponsored. It is supported by various organizations, including TechEquity Action and Legal Aid at Work. It is opposed by various industry associations, including Google and the American Council of Life Insurers. For a more thorough assessment, please see the Senate Judiciary Committee analysis.
Comments
According to the author:
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADT’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes
According to the Senate Appropriations Committee:
Unknown, potentially significant ongoing costs (General Fund) to the Civil Rights Division (CRD). The number of ADTs that this bill would regulate is unknown, but the number is surely very large and growing. Therefore, an unknown but significant number of complaints may be filed with the department under this bill, which CRD may prosecute. These prosecutions will involve advanced technologies and complex legal analyses and require highly-skilled investigators and attorneys working with technology and statistical experts. Should this bill be enacted, CRD would likely need a significant number of permanent positions and other resources, depending on the number and complexity of complaints filed with CRD. This would include funding for permanent positions across all of CRD’s divisions as well as funding to hire expert consultants.
SUPPORT: (Verified 8/16/24)
American Federation of Musicians, Local 7
California Employment Lawyers Association
Center for Democracy and Technology
Center on Race and Digital Justice Secure Justice
Consumer Reports
East Bay Community Law Center
Economic Security California Action
Equal Rights Advocates
The Greenlining Institute
Legal Aid at Work
Rise Economy
Techequity Collaborative
OPPOSITION: (Verified 8/16/24)
ACLU California Action
Advanced Medical Technology Association
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
America’s Physician Groups
Association of California Life & Health Insurance Companies
CalBroadband
California Association of Health Plans
California Bankers Association
California Communications Association
California Community Banking Network
California Credit Union League
California Financial Services Association
California Hospital Association
California Life Sciences
California Medical Association
California Mortgage Bankers Association
California Staffing Professionals
Consumer Technology Association
CTIA – The Wireless Association
Electronic Frontier Foundation
Kaiser Permanente
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Orange County Business Council
Pacific Association of Domestic Insurance Companies
Personal Insurance Federation of California
Sutter Health
USTelecom
Verizon Communications
ARGUMENTS IN SUPPORT: A coalition of groups, including Consumer Reports, Equal Rights Advocates, and the Greenlining Institute, write
AB 2930 would enact common-sense guardrails to help ensure that developers and deployers of these tools are obligated to test and mitigate for discriminatory outcomes prior to the sale or use of these tools in our communities. Specifically, the legislation would:
1. Require developers and deployers to conduct pre-deployment impact assessments to determine any potential for discrimination on people with protected class status;
2. Prohibit the sale or use of an ADT that may create a discriminatory outcome on people within a protected class until that adverse impact has been addressed and resolved;
3. Provide consumers with pre-use notice of the tool, a post-use explanation, the right to correct inaccurate information, and access to alternative selection procedures.
ARGUMENTS IN OPPOSITION: The Consumer Technology Association writes:
AB 2930 would require impact assessments be performed for “any” ADT the deployer uses, regardless of whether the use of the tool presents any significant risks. A risk-based approach to regulating AI tools is necessary to avoid overbroad regulations and costly new mandates that can, and should, be narrowly focused on only those use cases presenting greatest risks to individuals or society.
ASSEMBLY FLOOR: 50-14, 5/21/24
AYES: Addis, Aguiar-Curry, Alvarez, Arambula, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Wendy Carrillo, Connolly, Mike Fong, Garcia, Gipson, Grayson, Haney, Hart, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Ortega, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Rendon, Reyes, Santiago, Schiavo, Ting, Villapudua, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas
NOES: Alanis, Chen, Davies, Dixon, Flora, Vince Fong, Gallagher, Hoover, Lackey, Jim Patterson, Joe Patterson, Sanchez, Ta, Waldron
NO VOTE RECORDED: Bains, Juan Carrillo, Cervantes, Megan Dahle, Essayli, Friedman, Gabriel, Holden, Mathis, Ramos, Luz Rivas, Rodriguez, Blanca Rubio, Soria, Valencia, Wallis
Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113
8/19/24 10:55:55
**** END ****
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
AB 2930 (Bauer-Kahan) - Automated decision tools
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
AB 2930 would prohibit discrimination by automated decision tools (ADT), as specified.
*********** ANALYSIS ADDENDUM – SUSPENSE FILE ***********
The following information is revised to reflect amendments
adopted by the committee on August 15, 2024
Fiscal Impact:
Unknown, potentially significant ongoing costs (General Fund) to the Civil Rights Division (CRD). The number of ADTs that this bill would regulate is unknown, but the number is surely very large and growing. Therefore, an unknown but significant number of complaints may be filed with the department under this bill, which CRD may prosecute. These prosecutions will involve advanced technologies and complex legal analyses and require highly-skilled investigators and attorneys working with technology and statistical experts. Should this bill be enacted, CRD would likely need a significant number of permanent positions and other resources, depending on the number and complexity of complaints filed with CRD. This would include funding for permanent positions across all of CRD’s divisions as well as funding to hire expert consultants.
Committee Amendments:
- Revise the definition of “consequential decision” as “a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of employment with respect to all of the following: (1) Pay or promotion; (2) Hiring or termination; and (3) Automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment.”
- Delete all of the following from the definition of “consequential decision”:
- Education and vocational training, as specified;
- Housing or lodging, including rental or short-term housing or lodging;
- Essential utilities, as specified;
- Family planning;
- Adoption services, reproductive services, or assessments related to child protective services;
- Health care or health insurance, including mental health care, dental, or vision.
- Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor; and,
- Aspects of the criminal justice system, as specified.
- Education and vocational training, as specified;
- Remove “local government agency” from the definition of “Deployer”.
- Remove “state or local government agency” from the definition of “Developer”.
- Delete the provision defining “state government deployer.”
- Strike the provisions that require state government deployers to provide the California Privacy Protection Agency (PPA) a list of ADTs and that require the agency to identify a schedule for compliance.
- Delete “state government deployer” from the provision making it unlawful to retaliate against a natural person.
- Authorize the CRD to bring a civil action against a deployer or developer for a violation.
- Delete all provisions authorizing enforcement by district attorneys, county counsels, city attorneys, public attorneys, the PPA, the Attorney General.
- Delete provisions requiring deployers and developers to provide impact assessments to the PPA upon request, including provisions allowing the PPA to provide impact assessments to public entities, provisions relating to attorney client privilege and work product, disclosure of trade secrets, and the California public Records Act.
- Delete provisions establishing administrative fines.
-- END --
SENATE COMMITTEE ON APPROPRIATIONS
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
AB 2930 (Bauer-Kahan) - Automated decision tools
Version: July 3, 2024 | Policy Vote: JUD. 9 - 2 |
Urgency: No | Mandate: Yes |
Hearing Date: August 5, 2024 | Consultant: Liah Burnley |
Bill Summary:
AB 2930 would prohibit the use of automated decision tools (ADT) to make consequential decisions, if the deployer has identified a “reasonable risk” of “algorithmic discrimination,” unless the risk has been mitigated.
Fiscal Impact:
- Significant one-time and ongoing costs (General Fund) to the California Privacy Protection Agency (CPPA). CPPA indicates that, to implement AB 2930 as amended July 3, 2024, the CPPA anticipates costs associated with (1) establishing a schedule for state agency compliance with the requirements of the law; (2) setting up a system to receive impact assessments from covered entities, reviewing them, and transferring them to the enforcing entities; and (3) enforcing against covered entities for failure to submit the impact assessments as required. The CPPA estimates that AB 2930 will cost $3,001,997, in budget year (BY), $4,301,997 in BY1, and $2,028,797 in BY 2 and ongoing. CPPA anticipates that the agency will need 7 permanent positions to meet its responsibilities under the bill, including: 2.0 Attorney III, 1.0 Staff Services Manager I, 2.0 Associate Governmental Program Analyst, 1.0 Information Technology Specialist II, & 1.0 Information Technology Specialist I. The annual cost of the 7.0 positions will be approximately $ 1.2 million annually. Depending on the procurement vehicle the agency has to undergo the Project Approval Lifecycle (PAL), the agency’s cost estimate assumes it will go through the PAL process. (See Staff Comments)
- Significant ongoing costs (General Fund) to the Civil Rights Division (CRD). CRD notes that the number of ADTs that this bill would regulate is unknown, but the number is surely very large and growing. Therefore, CRD anticipates an unknown but significant number of complaints to be filed with the department under this bill, which CRD will be obligated to investigate, mediate, and potentially prosecute. These investigations, mediations, and prosecutions – because they will involve advanced technologies and complex legal analyses – will require highly-skilled investigators and attorneys working with technology and statistical experts. Among other activities, CRD will also need to conduct outreach and education and respond to public records requests. Should this bill be enacted, CRD anticipates requesting an unknown but significant number of permanent positions and other resources, costing in the low millions to low tens of millions annually, depending on the number and complexity of complaints filed with CRD. This would include funding for permanent positions across all of CRD’s divisions as well as funding to hire expert consultants.
- Significant ongoing costs to the state funded trial court system (General Fund, Trial Court Trust Fund). Judicial Council indicates that courts use several different ADTs for pretrial risk assessments, sentencing, and for determining court ordered debt. Notably, the MyCitations tool is used to provide recommendations to the court for determining reductions to court ordered debt for infraction violations. The courts also use ADTs to make recommendations for individual’s bail determinations. The Judicial Council estimates this bill will cost approximately $14.7 million in the first year of implementation. This includes $12.3 million for the trial courts’ data repository costs to enable data exchange and staffing and $2.4 million for the Judicial Council for staffing costs to conduct and produce the annual impact assessments for ADTs and implement a governance program. Total ongoing costs to implement this bill will be approximately $13.3 million annually. This includes $11 million for trial courts for ongoing data repository and staffing costs and $2.3 million for the Judicial Council for ongoing staffing costs to continue to produce annual impact assessments and to maintain the governance program. If funding is not provided for the new workload created by this bill, it could result in delays and prioritization of court cases and may impact access to justice. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations. The Budget Act of 2024, for the fiscal year beginning July 1, 2024, includes a $97 million reduction to the trial courts, a commensurate reduction of up to 7.95 percent to the budget for the state-level judiciary, and a reduction of the trial court state-level emergency reserve in the Trial Court Trust Fund from $10 million to $5 million. The Budget Act also includes a $37.3 million General Fund backfill for the Trial Court Trust Fund to address the continued decline in civil fee and criminal fine and penalty revenues expected in fiscal year 2024‒25.
- Significant ongoing costs (General Fund) to California Health and Human Services (CHHS). CHHS indicates that its Office of Technology and Solutions Integration (OTSI) estimates costs in the low millions to tens of millions, beginning at the time which PPA establishes a schedule for government compliance no later than January 1, 2027 and annually thereafter, with indeterminate but significant cost pressures related to assumed growth of these tools in the future. These costs are estimated and attributable due to legal and security resources for enforcement, compliance oversight, and IT development. This bill would impact many of the agency's information systems and data solutions, including California Department of Social Services (CDSS), Department of Rehabilitation (DOR), Department of Managed Health Care (DMHC), Health Care Access and Information (HCAI), and California Department of Aging (CDA), and would require programming and processing revisions, form revisions, and stakeholder engagement. The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the alternative processes that are yet to be determined.
- Unknown, significant possibly reimbursable ongoing costs (local funds, General Fund) to local agencies, to comply with the requirements of this bill, likely in the tens of millions, per agency, annually. Notably, this requirement applies broadly to every entity in this state. In the aggregate, ongoing costs may be in the billions of dollars. General Fund costs will depend on whether the duties imposed by this bill constitute a reimbursable state mandate, as determined by the Commission on State Mandates.
- Unknown, significant ongoing costs (General Fund) to state agencies, to comply with the requirements of this bill, likely in the tens of millions, per agency, annually. Notably, this requirement applies broadly to every entity in this state.
Background:
Automated decisionmaking is used across all industries. Transparency and fairness concerns have been raised about the use of ADTs to make determinations that have significant effects on human life.
Proposed Law:
- Requires deployers to perform an annual impact assessment on any ADT deployed, as specified.
- Provides that if the impact assessment identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the automated decision tool until the risk has been mitigated.
- Requires state and local agencies to provide a list of ADTs it deploys to the PPA, as specified, and requires the PPA to establish a staggered schedule that identifies when each agency shall comply with the provisions of this bill.
- Provides that prior to an ADT making a consequential decision, or being a substantial factor in making a consequential decision, a deployer shall notify any natural person that is subject to the consequential decision that an ADTs being used.
- Requires deployers and developers of an ADT to establish and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination, as specified. There must be at least one employee responsible for maintaining the program.
- Authorizes the CRD to investigate reports of algorithmic discrimination and other violations of this bill’s provisions.
- Authorizes PPA to bring enforcement actions against developers and deployers who violate this bill’s provisions, and establishes an administrative fine of not more than $10,000 for such violations.
- Authorizes the CRD, the Attorney General, district attorneys, county counsels, and city attorneys to bring a civil action to enforce these prohibitions and establishes statutory damages in the amount of $25,000 for claims involving algorithmic discrimination.
- Defines “Deployer” as a person, partnership, local government agency, developer, corporation, or any contractor or agent of those entities, that uses an ADT to make a consequential decision.
- Defines “Developer” as a person, partnership, state or local government agency, or corporation that designs, codes, or produces an ADT or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
- Defines “ADT” as an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
- Defines “Algorithmic discrimination” as the condition in which an automated decision tool contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
- Defines “Consequential decision” as a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, specified services and rights.
Related Legislation:
AB 331 (Bauer-Kahan), of this legislative session, also would have established an ADT regulation scheme. AB 331 was held on the Assembly Appropriations Committee’s suspense file.
Staff Comments:
It is unclear to what extent this bill could be interpreted to allow for greater discrimination by ADTs than allowable for human-made decisions. As some industry groups point out, this bill would permit deployers of ADTs to avoid meeting important obligations that are necessary to ensure that such tools, as they are used and implemented in practice, do not cause discrimination. Existing law outright prohibits unlawful discrimination on specified bases. (See, e.g. California’s Fair Employment Practices and Housing Act (FEHA) and the Unruh Civil Rights Act, etc.) This bill could allow otherwise unlawful discrimination when it is “algorithmic discrimination” and done by ADTS. Specifically, this bill allows a certain level of algorithmic discrimination, if a “reasonable risk” of discrimination (as determined by the ADT’s developer) is identified (by the developer) and “mitigated” (presumably, according to the developer’s standards). It is unclear what level of risk is reasonable for discrimination and how much of the risk needs to be mitigated? Accordingly, this bill attempts to resolve an issue at a significant state costs, that can likely already be enforced through the state’s existing anti-discrimination measures. Fortunately, due to state laws already on the books, entities cannot engage in unlawful discrimination, through humans or through their tools, including algorithms and ADTs.
This bill would significantly expand the scope of the CPPA. According to CPPA:
- First, AB 2930 as amended July 3, 2024 potentially invites uncertainty and inconsistency. There is a great deal of overlap between AB 2930 and the Agency’s draft automated decisionmaking technology (ADMT) and risk assessment regulations. Both the bill and the draft regulations generally require covered entities to perform risk assessments and provide notice and opt-out rights with respect to automated decision-making. Further, Proposition 24—which requires the agency to promulgate regulations on ADMT and risk assessments—only authorizes legislation that furthers the purposes of the Proposition, and considers any conflicting legislation null and void.
- If this bill were to be adopted in its current form, a separate and overlapping set of requirements would apply for businesses covered by both AB 2930 and the CCPA. This would cause a great deal of confusion and could make it difficult for businesses to comply. The problems will only get worse if AB 2930 is adopted and then amended in the future—creating the potential for further divergence and conflicting requirements. CPPA will likely incur litigation costs to sort out these uncertainties over the years to come, plus the internal costs of trying to harmonize these competing requirements.
- Enforcement: The CPPA’s Enforcement Division would need to create a separate workstream to contact businesses and evaluate impact assessments for referral to other agencies, resulting in additional costs. The Enforcement Division typically evaluates potential violations during the course of ongoing investigations under the CCPA. Thus, the Enforcement Division can address additional violations through an economy of scale with minimal additional resources. This bill, by contrast, would require the Enforcement Division to review impact assessments for non-compliance, and then refer the non-compliance to other agencies. It also requires that the Agency develop additional expertise in oversight of government entities, something currently outside the Agency’s remit.
- First, AB 2930 as amended July 3, 2024 potentially invites uncertainty and inconsistency. There is a great deal of overlap between AB 2930 and the Agency’s draft automated decisionmaking technology (ADMT) and risk assessment regulations. Both the bill and the draft regulations generally require covered entities to perform risk assessments and provide notice and opt-out rights with respect to automated decision-making. Further, Proposition 24—which requires the agency to promulgate regulations on ADMT and risk assessments—only authorizes legislation that furthers the purposes of the Proposition, and considers any conflicting legislation null and void.
Such a bifurcated process would complicate existing CCPA investigations, increase the burden on businesses who might need to engage with multiple regulators at the same time, and prevent the Enforcement Division from obtaining a complete remedy in a CCPA investigation. Although the Enforcement Division would not necessarily investigate a business for non-compliance with the impact assessment requirement and the CCPA at the same time in every case, the absence of full CPPA enforcement authority underscores the agency’s need to create separate workstreams. In addition, the similarity between the bill’s requirements and the CCPA’s regulations governing risk assessments would require coordination between agencies to reduce the number of overlapping investigations, potentially leading to inefficiency.
To manage and oversee these separate workstreams, the Enforcement Division would need to hire one Attorney III and one AGPA to provide administrative support. These needs might increase depending on the level of non-compliance, the complexity of the investigations, and the logistical difficulty of coordinating with multiple agencies for referrals.
An amendment to authorize, but not require, the CPPA to bring administrative enforcement actions for violation of the entire chapter, would streamline the CPPA’s existing investigations and allow the Enforcement Division to pursue holistic remedies on the same track. Although the Enforcement Division would still need to hire one Attorney III for litigation purposes, we would not need to hire the additional AGPA because we would not need to manage separate workstreams.
- Legal: In addition, the Legal Division would need an Attorney III and Staff Services Manager I to engage with all covered state departments required to prepare impact assessments, set their compliance schedule, and coordinate all impact assessment requests and transmissions to and from the various enforcement entities. These activities will require a significant amount of legal analysis and administrative management given the potentially large amount and diverse entities involved.
- IT: In addition, the Agency will require a secure file transfer system to receive, maintain, and transfer the impact assessments required by this bill. To facilitate the implementation of the bill as written, CPPA will need to quickly acquire a new secure system. This system will need to be able to facilitate regular and ad-hoc notifications, workflows, internal and external accounts for impacted entities. This system will need to able to securely request and contain highly sensitive information inside of the impact reports. In order to meet the January 1, 2026 deployment date, the Agency will require exemption from PAL and standard competitive procurements. The estimated time of the project would be for six months. The project team would consist of contracted resources from the system integration in addition to any development resources. This would be the bulk of the one-time costs. Ongoing costs will consist of an Information Technology Specialist II, Software licensing, CDT Services, and additional platform support.
- Legal: In addition, the Legal Division would need an Attorney III and Staff Services Manager I to engage with all covered state departments required to prepare impact assessments, set their compliance schedule, and coordinate all impact assessment requests and transmissions to and from the various enforcement entities. These activities will require a significant amount of legal analysis and administrative management given the potentially large amount and diverse entities involved.
Without an exemption from PAL, the law as written will not be completed until after the proposed legislative due dates. The estimated go-live would be January of 2027 after completion standard PAL and BCP timelines. Additional resources will be required to complete the necessary steps of PAL on-time.
-- END --
SENATE COMMITTEE ON
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
AB 2930 (Bauer-Kahan) - Automated decision tools
Version: | Policy Vote: |
Urgency: | Mandate: |
Hearing Date: | Consultant: |
Bill Summary:
AB 2930 would prohibit the use of automated decision tools (ADT) to make consequential decisions, if the deployer has identified a “reasonable risk” of “algorithmic discrimination,” unless the risk has been mitigated.
Fiscal Impact:
- Unknown, significant ongoing costs (General Fund) to the California Privacy Protection Agency (PPA) to receive a list from state and local agencies of ADTs to establish a staggered schedule that identifies when each agency shall comply with the provisions of this bill and bring administrative enforcement actions against developers and deployers who violate this bill’s provisions, likely in the tens of millions annually.
- Significant ongoing costs (General Fund) to the Civil Rights Division (CRD). CRD notes that the number of ADTs that this bill would regulate is unknown, but the number is surely very large and growing. Therefore, CRD anticipates an unknown but significant number of complaints to be filed with the department under this bill, which CRD will be obligated to investigate, mediate, and potentially prosecute. These investigations, mediations, and prosecutions – because they will involve advanced technologies and complex legal analyses – will require highly-skilled investigators and attorneys working with technology and statistical experts. Among other activities, CRD will also need to conduct outreach and education and respond to public records requests. Should this bill be enacted, CRD anticipates requesting an unknown but significant number of permanent positions and other resources, costing in the low millions to low tens of millions annually, depending on the number and complexity of complaints filed with CRD. This would include funding for permanent positions across all of CRD’s divisions as well as funding to hire expert consultants.
- Significant ongoing costs to the state funded trial court system (General Fund, Trial Court Trust Fund). Judicial Council indicates that courts use several different ADTs for pretrial risk assessments, sentencing, and for determining court ordered debt. Notably, the MyCitations tool is used to provide recommendations to the court for determining reductions to court ordered debt for infraction violations. The courts also use ADTs to make recommendations for individual’s bail determinations. The Judicial Council estimates this bill will cost approximately $14.7 million in the first year of implementation. This includes $12.3 million for the trial courts’ data repository costs to enable data exchange and staffing and $2.4 million for the Judicial Council for staffing costs to conduct and produce the annual impact assessments for ADTs and implement a governance program. Total ongoing costs to implement this bill will be approximately $13.3 million annually. This includes $11 million for trial courts for ongoing data repository and staffing costs and $2.3 million for the Judicial Council for ongoing staffing costs to continue to produce annual impact assessments and to maintain the governance program. If funding is not provided for the new workload created by this bill, it could result in delays and prioritization of court cases and may impact access to justice. While the courts are not funded on a workload basis, an increase in workload could result in delayed court services and would put pressure on the General Fund to fund additional staff and resources and to increase the amount appropriated to backfill for trial court operations. The Budget Act of 2024, for the fiscal year beginning July 1, 2024, includes a $97 million reduction to the trial courts, a commensurate reduction of up to 7.95 percent to the budget for the state-level judiciary, and a reduction of the trial court state-level emergency reserve in the Trial Court Trust Fund from $10 million to $5 million. The Budget Act also includes a $37.3 million General Fund backfill for the Trial Court Trust Fund to address the continued decline in civil fee and criminal fine and penalty revenues expected in fiscal year 2024‒25.
- Significant ongoing costs (General Fund) to California Health and Human Services (CHHS). CHHS indicates that its Office of Technology and Solutions Integration (OTSI) estimates costs in the low millions to tens of millions, beginning at the time which PPA establishes a schedule for government compliance no later than January 1, 2027 and annually thereafter, with indeterminate but significant cost pressures related to assumed growth of these tools in the future. These costs are estimated and attributable due to legal and security resources for enforcement, compliance oversight, and IT development. This bill would impact many of the agency's information systems and data solutions, including California Department of Social Services (CDSS), Department of Rehabilitation (DOR), Department of Managed Health Care (DMHC), Health Care Access and Information (HCAI), and California Department of Aging (CDA), and would require programming and processing revisions, form revisions, and stakeholder engagement. The costs to implement this bill depends on the number of solutions and complexity of developing and resourcing the alternative processes that are yet to be determined.
- Unknown, significant possibly reimbursable ongoing costs (local funds, General Fund) to local agencies, to comply with the requirements of this bill, likely in the tens of millions, per agency, annually. Notably, this requirement applies broadly to every entity in this state. In the aggregate, ongoing costs may be in the billions of dollars. General Fund costs will depend on whether the duties imposed by this bill constitute a reimbursable state mandate, as determined by the Commission on State Mandates.
- Unknown, significant ongoing costs (General Fund) to state agencies, to comply with the requirements of this bill, likely in the tens of millions, per agency, annually. Notably, this requirement applies broadly to every entity in this state.
Background:
Automated decisionmaking is used across all industries. Transparency and fairness concerns have been raised about the use of ADTs to make determinations that have significant effects on human life.
Proposed Law:
- Requires deployers to perform an annual impact assessment on any ADT deployed, as specified.
- Provides that if the impact assessment identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the automated decision tool until the risk has been mitigated.
- Requires state and local agencies to provide a list of ADTs it deploys to the PPA, as specified, and requires the PPA to establish a staggered schedule that identifies when each agency shall comply with the provisions of this bill.
- Provides that prior to an ADT making a consequential decision, or being a substantial factor in making a consequential decision, a deployer shall notify any natural person that is subject to the consequential decision that an ADTs being used.
- Requires deployers and developers of an ADT to establish and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination, as specified. There must be at least one employee responsible for maintaining the program.
- Authorizes the CRD to investigate reports of algorithmic discrimination and other violations of this bill’s provisions.
- Authorizes PPA to bring enforcement actions against developers and deployers who violate this bill’s provisions, and establishes an administrative fine of not more than $10,000 for such violations.
- Authorizes the CRD, the Attorney General, district attorneys, county counsels, and city attorneys to bring a civil action to enforce these prohibitions and establishes statutory damages in the amount of $25,000 for claims involving algorithmic discrimination.
- Defines “Deployer” as a person, partnership, local government agency, developer, corporation, or any contractor or agent of those entities, that uses an ADT to make a consequential decision.
- Defines “Developer” as a person, partnership, state or local government agency, or corporation that designs, codes, or produces an ADT or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
- Defines “ADT” as an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
- Defines “Algorithmic discrimination” as the condition in which an automated decision tool contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
- Defines “Consequential decision” as a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, specified services and rights.
Related Legislation:
AB 331 (Bauer-Kahan), of this legislative session, also would have established an ADT regulation scheme. AB 331 was held on the Assembly Appropriations Committee’s suspense file.
Staff Comments:
It is unclear to what extent this bill could be interpreted to allow for greater discrimination by ADTs than allowable for human-made decisions. As some industry groups point out, this bill would permit deployers of ADTs to avoid meeting important obligations that are necessary to ensure that such tools, as they are used and implemented in practice, do not cause discrimination. Existing law outright prohibits unlawful discrimination on specified bases. (See, e.g. California’s Fair Employment Practices and Housing Act (FEHA) and the Unruh Civil Rights Act, etc.) This bill could allow otherwise unlawful discrimination when it is “algorithmic discrimination” and done by ADTS. Specifically, this bill allows a certain level of algorithmic discrimination, if a “reasonable risk” of discrimination (as determined by the ADT’s developer) is identified (by the developer) and “mitigated” (presumably, according to the developer’s standards). It is unclear what level of risk is reasonable for discrimination and how much of the risk needs to be mitigated? Accordingly, this bill attempts to resolve an issue at a significant state costs, that can likely already be enforced through the state’s existing anti-discrimination measures. Fortunately, due to state laws already on the books, entities cannot engage in unlawful discrimination, through humans or through their tools, including algorithms and ADTs.
This bill would significantly expand the scope of the PPA. According to its website, the PPA’s mission is to protect consumer privacy. Neither algorithmic decisionmaking nor discrimination is within the purview of the PPA.
-- END --
Senator Thomas Umberg, Chair
2023-2024 Regular Session
Bill No: AB 2930 (Author:Bauer-Kahan)
Version: June 24, 2024
Hearing Date: July 2, 2024
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
Automated decision tools
DIGEST
This bill regulates the use of “automated decision tools” (ADTs) in order to prevent “algorithmic discrimination.” This includes requirements on developers and deployers that make and use these tools to make consequential decisions to perform impact assessments on ADTs. The bill establishes the right of individuals to know when an ADT is being used, the right to opt out of its use, and an explanation of how it is used.
EXECUTIVE SUMMARY
Owing to recent advances in processing power and the rise of big data, artificial intelligence’s (AI) capacity and the scope of its applications have expanded rapidly, impacting how we communicate, interact, entertain ourselves, travel, transact business, and consume media. It has been used to accelerate productivity and achieve efficiencies, but has also been used to constrain personal autonomy, compromise privacy and security, foment social upheaval, exacerbate inequality, spread misinformation, and subvert democracy.
Automated decisionmaking is one particular area where this technology is being increasingly deployed. Major transparency and fairness concerns have been raised about the use of ADTs to make consequential decisions, essentially determinations with significant legal or other material effect on one’s life. This bill seeks to regulate their use by both public and private actors by requiring impact assessments to evaluate their purpose, use of data, potential for bias, and the steps taken to address those risks. The bill also ensures that individuals that are subject to ADTs know when the tool is being used to make a “consequential decision” about them, are able to opt out of their use, and are given a reasonable explanation for the automated decision made and a chance to correct any incorrect data.
The bill is author-sponsored. It is supported by various organizations, including TechEquity Action and Legal Aid at Work. It is opposed by various industry associations, including Google and the American Council of Life Insurers.
PROPOSED CHANGES TO THE LAW
Existing law:
- Establishes the CCPA, which grants consumers certain rights with regard to their personal information, including enhanced notice, access, and disclosure; the right to deletion; the right to restrict the sale of information; and protection from discrimination for exercising these rights. It places attendant obligations on businesses to respect those rights. (Civ. Code § 1798.100 et seq.)
- Establishes the Consumer Privacy Rights Act (CPRA), which amends the CCPA and creates the California Privacy Protection Agency (PPA), which is charged with implementing these privacy laws, promulgating regulations, and carrying out enforcement actions. (Civ. Code § 1798.100 et seq.; Proposition 24 (2020).)
- Requires the Attorney General to adopt regulations governing access and opt-out rights with respect to businesses’ use of automated decisionmaking technology, including profiling and requiring businesses’ response to access requests to include meaningful information about the logic involved in those decisionmaking processes, as well as a description of the likely outcome of the process with respect to the consumer. (Civ. Code § 1798.185(a)(16).)
- Provides that beginning the later of July 1, 2021, or six months after the PPA provides notice to the Attorney General that it is prepared to begin rulemaking, the authority assigned to the Attorney General to adopt regulations under this section shall be exercised by the PPA. (Civ. Code § 1798.185(d).)
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act. (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civ. Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
- Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
- Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civ. Code § 3426.1(d).)
This bill:
- Requires a deployer to perform an impact assessment on any ADT before the tool is first deployed and annually thereafter. With respect to an ADT that a deployer first used prior to January 1, 2025, the deployer shall perform an impact assessment on that ADT before January 1, 2026, and annually thereafter.
- Provides, notwithstanding the above, that a deployer is not required to perform an impact assessment on an ADT before using it if all of the following are true:
- The deployer uses the ADT only for its intended use as determined by the developer of the ADT.
- The deployer does not make any substantial modifications to the ADT.
- The developer has performed any required impact assessment on the ADT.
- The developer of the ADT has provided documentation to the deployer, as specified.
- Requires a deployer to ensure that the above impact assessment includes all of the following:
- A statement of the purpose of the ADT and its intended benefits, uses, and deployment contexts.
- A description of specified features of the ADT, including the personal characteristics or attributes that the ADT will measure or assess, the method for doing so, and how they are relevant to the consequential decisions for which the ADT will be used, as well as information on its outputs.
- A summary of the categories of information collected from natural persons and processed by the ADT when it is used to make, or be a substantial factor in making, a consequential decision, including categories of sensitive information and information related to a natural person’s receipt of sensitive services.
- A statement of the extent to which the deployer’s use of the ADT is consistent with or varies from the statement required of the developer.
- An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of specified protected categories, resulting from the deployer’s use of the ADT.
- A description of the safeguards implemented, or that will be implemented, to address reasonably foreseeable risks of algorithmic discrimination that address all of the following:
- Whether the ADT could be modified to mitigate the risk of algorithmic discrimination.
- Whether effective accommodations can be provided for any limitations on accessibility.
- Whether less discriminatory procedures or methods could be employed to mitigate the risk of algorithmic discrimination.
- A description of how the ADT will be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
- A description of how the ADT has been or will be evaluated for validity, reliability, and relevance.
- Requires a developer, before making an ADT that it designs, codes, or produces available to potential deployers, to perform an impact assessment on the ADT and annually thereafter. For an ADT first made available before January 1, 2025, the developer shall perform an impact assessment before January 1, 2026, and annually thereafter. The impact assessment must include all of the following:
- A statement of the purpose of the ADT and its intended benefits, uses, and deployment contexts.
- A description of the ADT’s outputs and how they are used to make, or be a substantial factor in making, a consequential decision.
- A summary of the categories of information collected from natural persons and processed by the ADT in connection with consequential decisionmaking.
- An analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of specific protected categories resulting from the deployer’s use of the ADT.
- A description of the measures taken by the developer to mitigate the risk of algorithmic discrimination.
- A description of how the ADT can be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
- A description of how the ADT has been evaluated for validity, reliability, and relevance.
- Requires a deployer or developer to perform, as soon as feasible, an impact assessment with respect to a substantial modification to an ADT.
- Exempts deployers with fewer than 55 employees unless they used ADTs impacting more than 999 people during the previous calendar year.
- Requires a deployer, prior to an ADT making a consequential decision, or being a substantial factor in making a consequential decision, to notify any natural person that is subject to the consequential decision that an ADT is being used and to provide all of the following:
- A statement of the purpose of the ADT.
- Contact information for the deployer.
- A plain language description of the ADT that includes specified information, including the personal characteristics or attributes that the ADT will measure or assess, the methods by which it does so, and how those contribute to the ultimate consequential decision. The deployer must all disclose a summary of the most recent impact assessment and information on the ADT’s outputs, their format, structure, and how they are used.
- Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the ADT, as provided.
- Requires a deployer, if a consequential decision is made solely based on the output of an ADT, to, if technically feasible, accommodate a natural person’s request to not be subject to the ADT and to instead be subject to an alternative selection process or accommodation.
- Provides that after such a request a deployer may reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer is not obligated to provide the alternative.
- Requires a deployer that has deployed an ADT, to make, or be a substantial factor in making, a consequential decision concerning a natural person, to provide the person all of the following:
- A simple and actionable explanation that identifies the principal factors, characteristics, logic and other information related to the individual that led to the consequential decision.
- The role that the ADT played in the decisionmaking process.
- The opportunity to correct any incorrect personal data that the ADT processed in making, or as a substantial factor in making, the consequential decision.
- Requires the notices and other communications described above to meet specified conditions, including that they be in clear and plain language in specified languages.
- Requires a developer to provide a deployer with the results of any impact assessment performed on an ADT that the developer sells, licenses, or otherwise transfers to the deployer, along with documentation describing all the following:
- The intended uses and known limitations of the ADT, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
- The type of data used to program or train the ADT.
- How the ADT was evaluated for validity and explainability.
- The deployer’s responsibilities herein and any technical information necessary for a deployer to fulfill their obligations.
- States that the above does not require the disclosure of trade secrets, as defined in Section 3426.1 of the Civil Code.
- Requires a deployer or developer to establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an ADT, as specified. This program must be designed, as provided, including the designation of at least one employee to be responsible for overseeing and maintaining the governance program and overall compliance that has the authority to assert to the employee’s employer a good faith belief that the design, production, or use of an ADT fails to comply hereby and to complete assessments of any compliance issue raised by that employee. The program shall provide for annual and comprehensive reviews of policies, practices, and procedures to ensure compliance.
- Exempts from the preceding obligation deployers with fewer than 55 employees, unless the deployer used an ADT that impacted more than 999 people during the previous calendar year.
- Requires a deployer and developer to make publicly available, in a readily accessible manner, a clear policy that provides a summary of the types of ADTs currently in use or made available to others and how they manage the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the ADTs it currently uses or makes available to others.
- Provides that if an impact assessment performed by a deployer identifies a reasonable risk of algorithmic discrimination, the deployer shall not use the ADT until the risk has been mitigated. If an impact assessment performed by a developer identifies such a risk under deployment conditions reasonably likely to occur in this state, the developer shall not make the ADT available to potential deployers until the risk has been mitigated.
- Requires a state government deployer to provide, by January 1, 2026, the PPA a list of ADTs initially deployed prior to January 1, 2025, identifying:
- Each ADT deployed and the role of each in making consequential decisions.
- The population affected by each ADT.
- Requires the PPA, by January 1, 2027, to establish a staggered schedule that identifies when each state government deployer shall comply, prioritizing ADTs with the highest risk for algorithmic discrimination, including civil rights violations and other discriminatory outcomes. The schedule shall require full compliance by each state deployer by January 1, 2031 with deployers in violation subject to enforcement actions by the PPA.
- Authorizes the Civil Rights Department (CRD) to investigate a report of algorithmic discrimination or any other violation hereof.
- Requires a deployer or a developer, upon receiving a request from the PPA, to, within 30 days of the request, provide the PPA any impact assessment that it performed pursuant hereto. The disclosure does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist, and the assessment is exempt from the California Public Records Act.
- Provides that no provision herein shall be construed to require the disclosure of trade secrets, as defined.
- Subjects a deployer or developer who violates this subdivision to liability for an administrative fine of not more than $10,000 per violation in an administrative enforcement action brought by the PPA. Each day on which an ADT is used for which an impact assessment has not been submitted shall give rise to a distinct violation.
- Authorizes the PPA to provide an impact assessment it receives to specified public prosecutors or CRD to assist that entity in initiating or litigating a civil action.
- Provides that, in an action brought by those entities, a court may award injunctive and declaratory relief, as well as attorneys’ fees and costs. In an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation may also be awarded.
- Provides a 45-day right to cure to developers and deployers.
- Makes it unlawful for a deployer, state government deployer, or developer to retaliate against a natural person for that person’s exercise of rights provided herein.
- Exempts cybersecurity-related technology, including technology designed to detect, protect against, or respond to security incidents, identity theft, fraud, harassment, malicious or deceptive activities or any illegal activity, preserve the integrity or security of systems, or investigate, report, or prosecute those responsible for those actions, from its provisions.
- Clarifies that the rights, remedies, and penalties established herein are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws.
- Defines the relevant terms, including:
- “Algorithmic discrimination” means the condition in which an ADT contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
- “Automated decision tool” means an artificial intelligence system or service that makes a consequential decision, or is a substantial factor in making consequential decisions.
- “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, specified goods, services, and opportunities, including housing, employment, education, financial services, and specified aspects of the criminal justice system.
- “Deployer” means a person, partnership, local government agency, developer, corporation, or any contractor or agent of those entities, that uses an ADT to make a consequential decision.
- “Developer” means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an ADT, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party.
- “Substantial factor” means an element of a decisionmaking process that is capable of altering the outcome of the process.
- “Substantial modification” means a new version, new release, or other update to an ADT that materially changes its uses, intended uses, or outcomes.
- “Unlawful discrimination” means any act that violates Section 51 of the Civil Code, any act that constitutes an unlawful practice or unlawful employment practice, as specified, or any other practice or act that otherwise violates a state or federal law against discrimination.
COMMENTS
- Frameworks for responsible development and accountability in AI
As directed by the National AI Initiative Act of 2020, the National Institute of Standards and Technology (NIST) developed the AI Risk Management Framework to assist entities designing, developing, deploying, and using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. That framework highlights the serious risks at play and the uniquely challenging nature of addressing them in this context:
Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high or low-probability, systemic or localized, and high- or low-impact.
While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique. AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.
These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. [. . .]
AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.
With recent dramatic advances in the capabilities of AI systems, the need for such frameworks for accountability and responsible development have become ever more urgent. This is especially true with respect to ADTs. A number of examples of discriminatory ADTs have been provided by the author.
For instance, Amazon deployed ADTs for hiring purposes and provided an example of how bias can be built into ADTs:
Amazon.com Inc’s machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.
The team had been building computer programs since 2014 to review job applicants’ resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters.
Automation has been key to Amazon’s e-commerce dominance, be it inside warehouses or driving pricing decisions. The company’s experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars - much like shoppers rate products on Amazon, some of the people said.
“Everyone wanted this holy grail,” one of the people said. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way.
That is because Amazon’s computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.[1]
An additional example involved a discrimination charge by the Department of Housing and Urban Development against Meta, which emphasized the importance of knowing which characteristics are being considered by an ADT.[2] The settled claims involved Meta targeting users with “housing ads based on algorithms that relied partly on characteristics protected under the Fair Housing Act, like race, national origin and sex.” The charged also alleged that “Meta’s lookalike or special ad audience tool allowed advertisers to target users based on protected traits.”
In response to growing concerns about the increased deployment of ever-advanced ADTs, the Biden Administration published its Blueprint for an AI Bill of Rights, which is a set of five principles and associated practices to help guide the design, use, and deployment of AI to protect the rights of the American public:
- Safe and Effective Systems: You should be protected from unsafe or ineffective systems. Automated systems should be developed with consultation from diverse communities, stakeholders, and domain experts to identify concerns, risks, and potential impacts of the system.
- Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
- Data Privacy: You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used. You should be protected from violations of privacy through design choices that ensure such protections are included by default, including ensuring that data collection conforms to reasonable expectations and that only data strictly necessary for the specific context is collected. Designers, developers, and deployers of automated systems should seek your permission and respect your decisions regarding collection, use, access, transfer, and deletion of your data in appropriate ways and to the greatest extent possible; where not possible, alternative privacy by design safeguards should be used. Systems should not employ user experience and design decisions that obfuscate user choice or burden users with defaults that are privacy invasive. Consent should only be used to justify collection of data in cases where it can be appropriately and meaningfully given. Any consent requests should be brief, be understandable in plain language, and give you agency over data collection and the specific context of use; current hard-to-understand notice-and-choice practices for broad uses of data should be changed. Enhanced protections and restrictions for data and inferences related to sensitive domains, including health, work, education, criminal justice, and finance, and for data pertaining to youth should put you first. In sensitive domains, your data and related inferences should only be used for necessary functions, and you should be protected by ethical review and use prohibitions. You and your communities should be free from unchecked surveillance; surveillance technologies should be subject to heightened oversight that includes at least pre-deployment assessment of their potential harms and scope limits to protect privacy and civil liberties. Continuous surveillance and monitoring should not be used in education, work, housing, or in other contexts where the use of such surveillance technologies is likely to limit rights, opportunities, or access. Whenever possible, you should have access to reporting that confirms your data decisions have been respected and provides an assessment of the potential impact of surveillance technologies on your rights, opportunities, or access.
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.[3]
- Regulating ADTs based off of the Blueprint
This bill seeks to implement many of the principles laid out in the President’s blueprint in an effort to holistically regulate ADTs, which are defined as AI systems or services that make consequential decisions, or are a substantial factor in making consequential decisions. “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, access to, or the cost, terms, or availability of, specified goods, services, and opportunities, including housing, employment, education, financial services, and specified aspects of the criminal justice system.
According to the author:
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADT’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
- Impact assessments
In order to ensure ADTs are fair, transparent, and aligned with basic ethical standards, the bill requires developers and deployers to conduct impact assessments, laying out the details of the tool and an analysis of the risk of “algorithmic discrimination,” the ultimate target of the bill. “Algorithmic discrimination” means the condition in which an ADT contributes to unlawful discrimination, including differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state or federal law.
As stated in the President’s Blueprint:
Algorithmic Discrimination Protections: Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way. This protection should include proactive equity assessments as part of the system design, use of representative data and protection against proxies for demographic features, ensuring accessibility for people with disabilities in design and development, pre-deployment and ongoing disparity testing and mitigation, and clear organizational oversight.
Before making an ADT that it designs, codes, or produces available to potential deployers, a developer is required to perform such an assessment on the ADT and annually thereafter. The bill provides detailed specifications for what needs to be included in the impact assessment. This includes a statement of the purpose of the ADT and its intended benefits, uses, and deployment contexts and a description of the ADT’s outputs and how they are used to make, or be a substantial factor in making, a consequential decision. The impact assessment must also provide a summary of the categories of information collected from natural persons and processed by the ADT in connection with consequential decisionmaking.
Most importantly, the impact assessment must include an analysis of the risk of algorithmic discrimination, including adverse impacts on the basis of specified protected categories resulting from the deployer’s use of the ADT. A description of the measures taken by the developer to mitigate that risk and of how the ADT can be used by a natural person, or be monitored when it is used autonomously, to make, or be a substantial factor in making, a consequential decision.
The developer is then required to provide a deployer, the person or entity, including a governmental entity, that uses an ADT to make consequential decisions, with the results of any impact assessment performed on an ADT that the developer sells, licenses, or otherwise transfers to the deployer, along with documentation describing all the following:
- The intended uses and known limitations of the ADT, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use.
- The type of data used to program or train the ADT.
- How the ADT was evaluated for validity and explainability.
- The deployer’s responsibilities herein and any technical information necessary for a deployer to fulfill their obligations.
Deployers must also complete similar impact assessments, unless the following conditions are met:
- The deployer uses the ADT only for its intended use as determined by the developer.
- The deployer does not make any substantial modifications to the ADT.
- The developer has performed any required impact assessment on the ADT.
- The developer of the ADT has provided the above documentation to the deployer.
If those conditions are not met, the deployer must complete an impact assessment that includes much of what the developer is required to include. The assessment must also include a description of specified features of the ADT, including the personal characteristics or attributes that the ADT will measure or assess, the method for doing so, and how they are relevant to the consequential decisions for which the ADT will be used, as well as information on its outputs. It must also include a statement of the extent to which the deployer’s use of the ADT is consistent with or varies from the statement required of the developer.
These impact assessments must be performed annually and, if there is a substantial modification made to the ADT, as soon as feasible thereafter. Where the assessment identifies a reasonable risk of algorithmic discrimination, the developer shall not make it available to deployers, and the deployer shall not use the ADT, until the risk has been mitigated.
A deployer or developer must also establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards designed to map, measure, and manage the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an ADT, as specified. This requires designation of at least one employee to be responsible for overseeing and maintaining the governance program and overall compliance with the provisions of this bill. The program shall all provide for annual and comprehensive reviews of policies, practices, and procedures to ensure compliance.
To lessen the impact on small businesses, the above requirements do not apply to deployers with fewer than 55 employees unless their use of ADTs impacts 1000 people or more.
Deployers that are state government agencies must provide the PPA a list of ADTs deployed, the role they have in making consequential decisions, and the populations thereby affected. The PPA is directed to establish a schedule for agency compliance.
- Notice, disclosures, and the right to opt out for individuals subjected to ADTs
Again, the Blueprint provides:
- Notice and Explanation: You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you. Designers, developers, and deployers of automated systems should provide generally accessible plain language documentation including clear descriptions of the overall system functioning and the role automation plays, notice that such systems are in use, the individual or organization responsible for the system, and explanations of outcomes that are clear, timely, and accessible. Such notice should be kept up-to-date and people impacted by the system should be notified of significant use case or key functionality changes. You should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.
- Human Alternatives, Consideration, and Fallback: You should be able to opt out from automated systems in favor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable expectations in a given context and with a focus on ensuring broad accessibility and protecting the public from especially harmful impacts.
This bill effectuates these principles by requiring a deployer, prior to an ADT making a consequential decision, or being a substantial factor in making a consequential decision, to notify any natural person that is subject to the consequential decision that an ADT is being used. That person shall be provided all of the following:
- A statement of the purpose of the ADT.
- Contact information for the deployer.
- A plain language description of the ADT that includes specified information, including the personal characteristics or attributes that the ADT will measure or assess, the methods by which it does so, and how those contribute to the ultimate consequential decision. The deployer must all disclose a summary of the most recent impact assessment and information on the ADT’s outputs, their format, structure, and how they are used.
- Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the ADT, as provided.
Furthermore, the bill requires a deployer, if a consequential decision is made solely based on the output of an ADT, to accommodate a natural person’s request to not be subject to the ADT and to instead be subject to an alternative selection process or accommodation However, to mitigate the impact on businesses this is only required if technically feasible and if the person does not provide specified information, the deployer is not so obligated.
Once the ADT is involved in making a consequential decision, the deployer is required to provide the person all of the following:
- A simple and actionable explanation that identifies the principal factors, characteristics, logic and other information related to the individual that led to the consequential decision.
- The role that the ADT played in the decisionmaking process.
- The opportunity to correct any incorrect personal data that the ADT processed in making, or as a substantial factor in making, the consequential decision.
This provides transparency and some measure of control to the subject of the ADT.
The required notices and other communications must meet specified conditions, including that they be in clear and plain language in specified languages.
The bill makes it unlawful for a deployer, state government deployer, or developer to retaliate against a natural person for exercising any rights provided herein.
- Enforcement and oversight
The PPA is the primary regulatory and enforcement entity for implementing the CCPA and CPRA. The PPA is currently in the process of drafting regulations to govern the use of automated decisionmaking technology by CCPA-covered businesses.
This bill requires deployers and developers to provide the PPA with any impact assessment performed within 30 days of a request. The PPA is authorized to bring an administrative enforcement action against a developer or deployer in violation of these provisions seeking an administrative fine of up to $10,000 per violation. CRD is explicitly granted authority to investigate a report of algorithmic discrimination or any other violation of the bill.
The bill further provides that CRD, the Attorney General, district attorneys, county counsel, city attorneys, and certain city prosecutors, as specified, may bring a civil action against a developer or deployer in violation seeking injunctive and declaratory relief, as well as attorney’s fees and costs. Where the violation involves algorithmic discrimination, the public prosecutors may also seek a civil penalty of $25,000 per violation. To assist these entities in such enforcement actions, the PPA is authorized to provide them with any impact assessments it receives. However, before initiating such actions, the public prosecutor must provide the developer or deployer in alleged violation with 45-days written notice and an opportunity to cure. If the violation is cured and a written notice attesting to such cure is provided, no claim for injunctive relief may be maintained.
- Other jurisdictions
A number of jurisdictions have stepped forward to respond to the dramatic increase in ADT usage. For instance the European Union AI Act provides guardrails for what it deems “high-risk AI systems” to ensure transparency and fairness. Here in the United States, a number of states have introduced legislation in this space, including New York and Connecticut. However, the first comprehensive state-level regulation has come in Colorado.
The Colorado law, approved by their Governor on May 17, 2024, places requirements on developers and deployers to use reasonable care to protect consumers from the risks of algorithmic discrimination. Many provisions of the Colorado law are found in this bill.
- Stakeholder engagement and positions
A number of stakeholders from industry associations, to labor groups, to public-interest advocacy organizations have engaged with the author and this Committee on the bill. A broad set of amendments recently made to the bill respond to a number of the concerns that have been raised by various groups and represent a reasonable compromise. For instance, recent amendments replaced “unjustified differential treatment or impacts” with “unlawful discrimination” in response to concerns from businesses that the former wording lacked clarity.
One additional change was an expansion of what materials were exempt from disclosure. Formerly, only trade secrets were not subject to disclosure pursuant to the California Public Records Act. Recent amendments make all impact assessments disclosed to the PPA exempt. The bill still makes clear that nothing therein requires the disclosure of trade secrets. However, claiming exemptions for trade secrets is a heavily litigated subject with an over reliance on such clauses. That is what led Colorado to specifically provide in its law that where a developer or deployer withholds information on this basis, it must notify the relevant party, including the consumer about the withholding and the basis for it. The author has agreed to amendments that place similar protections into the bill.
Another recent amendment fleshes out an existing exemption for cybersecurity-related technology, including technology designed to detect, protect against, or respond to security incidents, identity theft, fraud, harassment, malicious or deceptive activities or any illegal activity, preserve the integrity or security of systems, or investigate, report, or prosecute those responsible for those actions. The corresponding exemption in the Colorado law, however, specifically caveats a similar exemption to the technology by extending to it unless deployed to make, or be a substantial factor in making, a consequential decision. The author may wish to consider amendments that provide a similar caveat to ensure there are no loopholes in the protections provided for by the bill.
The recent changes have brought a number of groups from a support-if-amended position into support. A coalition of groups, including Consumer Reports, Equal Rights Advocates, and the Greenlining Institute, write in support:
While the advent of generative AI and large language models has been a new piece of the puzzle, ADTs have long existed in our communities. ADTs have been woven into the daily lives of our community members—increasingly these tools are dictating access to and the quality of housing, employment, credit, and many other critical services Californians need. The potential harms of these systems have also been well documented—from predictive models on recidivism that have been found to incorrectly predict that Black defendants are two times more likely to recidivate than white defendants to the use of algorithms to determine outcomes for family separations resulting in a potentially disparate impact on disabled parents and children. These tools have also been integrated into private businesses’ enterprise systems and used by governments to dictate who receives unemployment insurance or to determine who may get access to affordable housing.
AB 2930 would enact common-sense guardrails to help ensure that developers and deployers of these tools are obligated to test and mitigate for discriminatory outcomes prior to the sale or use of these tools in our communities. Specifically, the legislation would:
1. Require developers and deployers to conduct pre-deployment impact assessments to determine any potential for discrimination on people with protected class status;
2. Prohibit the sale or use of an ADT that may create a discriminatory outcome on people within a protected class until that adverse impact has been addressed and resolved;
3. Provide consumers with pre-use notice of the tool, a post-use explanation, the right to correct inaccurate information, and access to alternative selection procedures.
Various organizations have written requesting amendments to narrow the scope of the bill. Writing in a support if amended position, the California Nurses Association requests amendments to exclude post-hiring employment-related ADTs. Additionally, a number of associations representing various sectors have written in opposition to the bill and are requesting exemptions for their respective industries. This includes separate coalitions representing health care entities, the finance industry, and the insurance industry. The general argument is that other laws already adequately regulate these industries and the additional obligations imposed by this bill will disrupt operations. The Consumer Technology Association also writes in opposition that the focus on all ADTs is overbroad:
AB 2930 would require impact assessments be performed for “any” ADT the deployer uses, regardless of whether the use of the tool presents any significant risks. A risk-based approach to regulating AI tools is necessary to avoid overbroad regulations and costly new mandates that can, and should, be narrowly focused on only those use cases presenting greatest risks to individuals or society.
It should be noted that the bill makes clear that it is cumulative to existing law. Therefore, the bill does not impact any other legal requirements, rights, or remedies provided under existing law. For instance, nothing therein should be interpreted to impact existing anti-discrimination laws or the enforcement of them, and compliance with the provisions of this bill are not relevant to actions brought pursuant to other laws.
Writing in opposition, Google argues:
Section 22576(a) currently provides that “‘Algorithmic discrimination’ means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.” This broad definition of algorithmic discrimination advances a novel legal concept of algorithmic discrimination that has no clear contours and is untethered from unlawful discrimination under California’s existing civil rights laws.
We respectfully request that at minimum AB 2930 be amended to replace the word “unjustified” with “unlawful” and “contributes to” with “results in” in the definition of algorithmic discrimination. Section 22576(a).[4]
More generally, the bill should not create a two-tiered standard for illegal discrimination with one for human decisionmakers and another for software. Unless “algorithmic discrimination” is synomous [sic] with “unlawful discrimination”, Section 22756.6’s prohibitions on using and making available ADTs resulting in algorithmic discrimination would create a new category of illegal discrimination. Even if the bill were amended to add “currently unlawful” to the definition of algorithmic discrimination, the provision would create confusion by adding a new enforcement mechanism.
We believe that if it is illegal to do something without AI, it is illegal to do it with AI. If there is concern that existing California civil rights statutes do not explicitly state that they apply to AI or ADTs, the better approach would be to add clarifying language to existing laws. These laws can require safe and responsible development and deployment of ADTs for purposes of preventing illegal discrimination and punishing the failure to satisfy impact assessment, governance, and transparency requirements. Existing law should continue to prohibit and punish acts resulting in unlawful discrimination.
SUPPORT
American Federation of Musicians, Local 7
California Employment Lawyers Association
Center for Democracy and Technology
Center on Race and Digital Justice Secure Justice
Consumer Reports
East Bay Community Law Center
Economic Security California Action
Equal Rights Advocates
The Greenlining Institute
Legal Aid at Work
Rise Economy
Techequity Collaborative
OPPOSITION
ACLU California Action
Advanced Medical Technology Association
American Council of Life Insurers
American Property Casualty Insurance Association
America’s Physician Groups
Association of California Life & Health Insurance Companies
California Association of Health Plans
California Bankers Association
California Community Banking Network
California Credit Union League
California Financial Services Association
California Hospital Association
California Life Sciences
California Medical Association
California Mortgage Bankers Association
Consumer Technology Association
Electronic Frontier Foundation
Kaiser Permanente
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Orange County Business Council
Pacific Association of Domestic Insurance Companies
Personal Insurance Federation of California
Sutter Health
Verizon Communications
RELATED LEGISLATION
Pending Legislation:
SB 892 (Padilla, 2024) requires the California Department of Technology (CDT) to develop and adopt regulations to create an AI risk management standard, as specified. It requires the AI risk management standard to include, among other things, a detailed risk assessment procedure for procuring automated decision systems (ADS), as defined, that analyzes specified characteristics of the ADS, methods for appropriate risk controls, as provided, and adverse incident monitoring procedures. It requires CDT to, among other things, collaborate with specified organizations to develop the AI risk management standard. SB 892 is currently in the Assembly Privacy and Consumer Protection Committee.
AB 2885 (Bauer-Kahan, 2024) establishes a uniform definition for “artificial intelligence” in California’s codes. AB 2885 is currently in the Senate Appropriations Committee.
Prior Legislation:
AB 302 (Ward, Ch. 800, Stats. 2023) requires CDT, on or before September 1, 2024, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
AB 331 (Bauer-Kahan, 2023) was substantially similar to the current bill. AB 331 died in the Assembly Appropriations Committee.
PRIOR VOTES:
Assembly Floor (Ayes 50, Noes 14)
Assembly Appropriations Committee (Ayes 11, Noes 4)
Assembly Judiciary Committee (Ayes 9, Noes 2)
Assembly Privacy and Consumer Protection Committee (Ayes 8, Noes 3)
**************
Jeffrey Dastin, Insight - Amazon scraps secret AI recruiting tool that showed bias against women (October 10, 2018) Reuters, https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G/. All internet citations are current as of June 30, 2024. ↑
Lauren Feiner, DOJ settles lawsuit with Facebook over allegedly discriminatory housing advertising (June 21, 2022) CNBC, https://www.cnbc.com/2022/06/21/doj-settles-with-facebook-over-allegedly-discriminatory-housing-ads.html. ↑
Blueprint For An AI Bill Of Rights (October 2022) Office of Science and Technology Policy, https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf. ↑
As referenced above, some of these concerns have been addressed by amendments recently taken. ↑
ASSEMBLY THIRD READING
ATR Bill Id:AB 2930
Author:(Bauer-Kahan)
As Amended Ver:May 16, 2024
Majority vote
SUMMARY
Establishes a statutory framework to further the safe and informed use of automated decision tools (ADTs) in California.
Major Provisions
- Requires that developers and deployers of ADTs:
- Perform impact assessments before first using an ADT and annually thereafter.
- Perform impact assessments if an ADT they develop or deploy receives a significant update.
- Establish governance programs, maintained by at least one employee, that assess and address risks of algorithmic discrimination arising from the use of ADTs.
- Make publicly available documentation related to the types of ADTs currently used or made available by the entity, as well as how the entity manages algorithmic discrimination risks arising from the use of these ADTs.
- Provide impact assessments to the Civil Rights Department (CRD) upon request.
- Requires that deployers of ADTs:
- Notify individuals that are subject to a consequential decision made by an ADT, and provide subject individuals with relevant information.
- Accommodate an individual's request to not be subject to an ADT’s decision, if technically feasible, and provide an alternate process.
- Requires a developer of ADTs to:
- Provide a deployer with a statement regarding the intended uses of an ADT they provide, along with documentation relating to the ADT’s known limitations, training data, and how the tool was evaluated for validity and explainability. Exempts disclosure of trade secrets.
- Prohibits deployers and developers from using or making available discriminatory ADTs.
- Permits CRD to:
- Investigate reports of algorithmic discrimination.
- Bring an administrative enforcement action not exceeding $10,000 per violation against any developer or deployer who violates the requirement to disclose impact assessments.
- Establish a staggered schedule for compliance of the bill’s provisions by state government deployers of ADTs.
- Adopt rules and regulations to effectuate the bill's provisions.
- Permits the Attorney General, district attorneys, a county counsel, city attorneys, and city prosecutors to bring a civil action against a developer or deployer for a violation of the bill. Permits a court to award injunctive relief, declaratory relief, reasonable attorney's fees and litigation costs, and, in cases where a violation involves algorithmic discrimination, a civil penalty of $25,000 per violation. Requires a developer or deployer to be provided with 45 days written notice before an action is commenced against them, during which they can cure the noticed violation.
COMMENTS
Background. The term "ADT" refers to artificial intelligence systems that make, or are a substantial factor in making, consequential decisions. While using an ADT is often beneficial, ADTs are far from flawless. Many aspects of the real world are not reducible to data, and many complex situations present competing demands that human beings remain best equipped to assess. Furthermore, if the data used to train an ADT is biased, the tool's outputs will be similarly biased. AI-powered ADTs are increasingly influencing the lives of Californians. These tools are used to assess eligibility for loans and jobs, and influencing outcomes in healthcare and criminal justice. These tools can, when poorly designed and improperly used, undermine privacy and equity. Without a robust legal framework for their use, ADTs risk perpetuating systemic biases through opaque decision-making and a general lack of accountability.
What this bill would do. This bill would establish requirements for informed, conscious deployment of ADTs. Much of the bill is devoted to ensuring that ADT developers and deployers thoroughly consider the design, testing, and functioning of ADTs before releasing them into the world. The bill explicitly prohibits deployers and developers from using or making available ADTs that result in discriminatory outcomes.
Analysis. This bill seeks to tackle the risks associated with ADTs by enforcing transparency and ensuring that decision-making tools are scrutinized and pressure-tested for discriminatory outcomes. Such an approach safeguards fundamental rights and fosters trust in the technologies that are reshaping society.
According to the Author
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADT’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes…As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
Arguments in Support
Taking a "support if amended" position, a coalition of labor and equity organizations that includes TechEquity Action writes:
California has the opportunity to lead the nation with innovative policy that places common-sense guardrails on the development and use of AI and automated decision-making tools. Legislating on emerging technology requires thoughtful and deliberative work, and engagement with all the stakeholders who will be impacted. We commend you for taking on this challenge.
Arguments in Opposition
California Chamber of Commerce writes:
We agree that it is critical that companies take care to reduce bias and discrimination in consequential decisions impacting people. At the same time, we believe these issues exist whether these decisions are human made from start to finish, or a biproduct of using or incorporating new technologies in the decision-making process. As such, it is critical that any regulatory efforts proceed with caution and precision, particularly as technology is still developing and has the potential to reduce, if not one day eliminate, such undesirable outcomes.
FISCAL COMMENTS
According to Assembly Appropriations Committee:
1) Costs (General Fund, special funds) of an unknown but significant amount to each qualifying state agency that deploys ADTs to make consequential decisions. These agencies ("state government deployers") must provide CRD with a list of applicable ADTs and, by January 1, 2031, come into compliance with the bill’s other requirements, as specified in the schedules established by CRD. In the aggregate for all state government deployers, costs to come into compliance may be in the hundreds of millions of dollars. State government deployers will also incur ongoing costs of an unknown but significant amount to continue to comply with the bill's requirements.
2) Costs (General Fund) to CRD of an unknown but significant amount, possibly in the low tens of millions of dollars annually. CRD will incur workload costs in establishing and enforcing compliance schedules for state government deployers. CRD anticipates it will require significant additional staffing to review and respond to complaints alleging ADT violations, conduct investigations, and enforce meritorious claims against deployers. CRD reports it will need highly-skilled investigators and attorneys, as well as technology and statistical experts, to fulfill its obligations under the bill. Finally, CRD notes it is not clear whether penalties assessed by the department could or should be used to offsets the department's costs.
3) Possible costs to the Department of Justice (DOJ) to represent CRD in appellate litigation. DOJ expects "unquantifiable, but potentially significant" costs if CRD brings cases that DOJ must handle on appeal. These costs would be recovered by billing the client agency (CRD) but may impact the Legal Services Revolving Fund. Additionally, if DOJ chooses to bring its own enforcement actions, as authorized by this bill, DOJ may incur significant costs of an unknown amount (General Fund).
4) Costs (Trial Court Trust Fund, General Fund) of an unknown but potentially significant amount to the courts to adjudicate enforcement actions authorized by this bill. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The Governor's 2024-25 budget proposes $83.1 million ongoing from the General Fund to backfill declining revenue to the Trial Court Trust Fund.
VOTES
ASM PRIVACY AND CONSUMER PROTECTION: 8-3-0
YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson
NO: Joe Patterson, Dixon, Hoover
ASM JUDICIARY: 9-2-1
YES: Kalra, Bauer-Kahan, Bryan, Connolly, Haney, Maienschein, McKinnor, Pacheco, Reyes
NO: Dixon, Sanchez
ABS, ABST OR NV: Waldron
ASM APPROPRIATIONS: 11-4-0
YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart, Pellerin, Villapudua
NO: Sanchez, Dixon, Jim Patterson, Ta
UPDATED
VERSION: May 16, 2024
CONSULTANT: Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0003409
Date of Hearing: May 8, 2024
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
ABPCA Bill Id:AB 2930
Author:(Bauer-Kahan) – As Amended Ver:April 24, 2024
Policy Committee: | Privacy and Consumer Protection | Vote: | 8 - 3 |
Judiciary | 9 - 2 | ||
|
|
Urgency: No State Mandated Local Program: Yes Reimbursable: No
SUMMARY:
This bill implements a regulatory framework for use of automated decision tools (ADTs), with the goal of preventing algorithmic discrimination, and authorizes the Civil Rights Department (CRD) and specified public prosecutors to enforce violations.
Specifically, among other provisions, the bill defines a qualifying agency as a state or local agency with 25 or more employees that deploys ADT for consequential decision-making that impacts more than 999 people per year. The bill’s requirements also apply to private-sector entities of the same size that deploy ADT for consequential decision-making.
Consequential decision-making includes decisions related to employment, education and vocational training, essential utilities, adoption, health care, transportation, and criminal justice, among other subjects specified in the bill.
The bill requires a qualifying entity to perform an annual impact assessment of its ADT, make required notifications to people affected by consequential ADT decision-making, process and where feasible comply with opt-out requests, maintain a governance system, and make summary ADT information available to the public, among the bill’s other requirements. A qualifying deployer’s annual impact assessment must include an analysis of potential adverse impacts of the deployer’s use of ADT to people with legally protected characteristics and the safeguards implemented by the deployer to mitigate those potential adverse impacts.
FISCAL EFFECT:
- Costs (General Fund, special funds) of an unknown but significant amount to each qualifying state agency that deploys ADTs to make consequential decisions and must comply with the bill’s requirements, summarized above. By way of illustration of state agency costs, Judicial Council estimates annual ongoing costs of at least $12 million to comply with the bill. The California Department of Health and Human Services, which has 12 departments statewide, estimates costs in the low millions to tens of millions of dollars annually, with indeterminate but significant cost pressures related to assumed growth of these tools in the future. In the aggregate for all qualifying state agencies, costs may be in the hundreds of millions of dollars annually ongoing.
- Likely non-reimbursable costs to qualifying local agencies to comply with all of the above-described requirements, possibly in the tens of millions to hundreds of millions of dollars annually statewide.
- Costs (General Fund) to CRD of an unknown but significant amount, possibly in the low tens of millions of dollars annually. CRD anticipates it will require significant additional staffing to review and respond to complaints alleging ADT violations, conduct investigations, and enforce meritorious claims against deployers. CRD reports it will need highly-skilled investigators and attorneys, as well as technology and statistical experts, to fulfill its obligations under the bill. Finally, CRD notes it is not clear whether penalties assessed by the department could or should be used to offsets the department’s costs.
- Possible costs to the Department of Justice (DOJ) to represent CRD in appellate litigation. DOJ expects “unquantifiable, but potentially significant” costs if CRD brings cases that DOJ must handle on appeal. These costs would be recovered by billing the client agency (CRD) but may impact the Legal Services Revolving Fund. Additionally, if DOJ chooses to bring its own enforcement actions, as authorized by this bill, DOJ may incur significant costs of an unknown amount (General Fund).
- Costs (Trial Court Trust Fund, General Fund) of an unknown but potentially significant amount to the courts to adjudicate enforcement actions authorized by this bill. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a need for increased funding for courts from the General Fund. The Governor’s 2024-25 budget proposes $83.1 million ongoing from the General Fund to backfill declining revenue to the Trial Court Trust Fund.
According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years.
COMMENTS:
- Purpose. According to the author:
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADT’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes…As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
- Background. ADTs, and artificial intelligence more generally, are used ever more frequently, and in ever more sophisticated ways, to help companies and government agencies make decisions. In 2023, the White House released its Blueprint for an AI Bill of Rights, stating:
Among the great challenges posed to democracy today is the use of technology, data, and automated systems in ways that threaten the rights of the American public. Too often, these tools are used to limit our opportunities and prevent our access to critical resources or services. These problems are well documented. In America and around the world, systems supposed to help with patient care have proven unsafe, ineffective, or biased. Algorithms used in hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed new harmful bias and discrimination. Unchecked social media data collection has been used to threaten people’s opportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or consent.
This bill establishes a state regulatory framework of ADTs that complements the federal blueprint and prohibits the use and distribution of ADTs that result in algorithmic discrimination. The bill gives CRD the authority to investigate complaints of algorithmic discrimination, and allows CRD and other specified public prosecutors to file enforcement actions for any violation.
- Related Legislation. AB 331 (Bauer-Kahan), of this legislative session, also establishes an ADT regulation scheme. As described in the analysis of this bill by the Assembly Committee on Judiciary, AB 331 is different than AB 2930 in several ways, but generally applies to the same deployers. AB 331 was held on this committee’s suspense file.
Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081
Date of Hearing: April 23, 2024
ASSEMBLY COMMITTEE ON JUDICIARY
Ash Kalra, Chair
ABPCA Bill Id:AB 2930
Author:Bauer-Kahan – As Amended Ver:April 22, 2024
As Proposed to be Amended
SUBJECT: Automated decision tools
KEY ISSUE: SHOULD DEVELOPERS AND DEPLOYERS BE REQUIRED TO COMPLETE ANNUAL IMPACT ASSESSMENTS FOR AUTOMATED DECISION TOOLS, (adts) COMPLY WITH SPECIFIED NOTICE PROVISIONS UPON USING ADTS, AND BE PROHIBITED FROM USING OR PROVIDING ADTS THAT RESULT IN ALGORITHMIC DISCRIMINATION?
SYNOPSIS
For years, automated decision tools (ADT) (also known as automated systems, or automated decision technology) have been integrated into our lives. Briefly defined, ADTs are technological tools that make individualized decisions based on a coding structure. While the code and development of the automated technology is manmade, the ultimate decision is theoretically void of any human input. Most of us take automated technology and its algorithms for granted, often never recognizing when they are impacting our day-to-day lives. While this ensures that we can move about our lives unimpeded, it also means that we are often unaware of the prevalence of this relatively new technology. Like most any new development, automated technology has both positive and negative ramifications. This bill proposes a number of requirements for deployers and developers, as defined, of ADTs; prohibits the use of ADTs that result in algorithmic discrimination, and of particular relevance to our Committee, implements an enforcement scheme to promote the bill’s underlying goals. The author proposes to amend the bill to authorize the Civil Rights Department (CRD) to develop rules and regulations to effectuate their administrative enforcement authorization. This amendment is incorporated into the SUMMARY and COMMENTS sections of this analysis.
This bill is supported by SecureJustice. It is opposed by a coalition of business trade associations, including the California Chamber of Commerce. A broad coalition of organizations, including Tech Equity, Oakland Privacy, and SEIU California, takes a support if amended position. Internet.Works has submitted a position of oppose unless amended, and the Alliance for Digital Innovation and the California Radiological Society have submitted letters of concern. This bill was previously heard and approved by the Assembly Committee on Privacy and Consumer Protection by a vote of 8-3.
SUMMARY: Establishes a framework for the regulation of automated decision technology. Specifically, this bill:
- Defines the following for purposes of the bill:
- “Algorithmic discrimination” means the condition in which an automated decision tool contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law;
- “Artificial intelligence” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments;
- “Automated decision tool” means a system or service that uses artificial intelligence and has been specifically developed to, or specifically modified to, make, or be a substantial factor in making, consequential decisions;
- “Consequential decision” means a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of, any of the following:
- Employment with respect to all of the following: pay or promotion, hiring or termination, automated task allocation that limits, segregates, or classifies employees for the purpose of assigning or determining material terms or conditions of employment;
- Education and vocational training as it relates to all of the following: assessment, detecting student cheating or plagiarism, accreditation, certification, admissions, financial aid or scholarships;
- Housing or lodging, including rental or short-term housing or lodging;
- All of the following essential utilities: electricity, heat, water, internet or telecommunications access, transportation;
- Family planning;
- Adoption services, reproductive services, or assessments related to child protective services;
- Health care or health insurance, including mental health care, dental, or vision;
- Financial services, including a financial service provided by a mortgage company, mortgage broker, or creditor;
- Aspects of the criminal justice system: risk assessments for pretrial hearings, sentencing, parole;
- Legal services;
- Private arbitration;
- Mediation;
- Voting.
- “Deployer” means a person, partnership, state or local government agency, developer, or corporation that uses an automated decision tool to make a consequential decision;
- “Developer” means a person, partnership, state or local government agency, or corporation that designs, codes, or produces an automated decision tool, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a substantial factor in making, consequential decisions, whether for its own use or for use by a third party;
- “Impact assessment” means a documented risk-based evaluation of an automated decision tool that means criteria specified in 3);
- “Sex” includes pregnancy, childbirth, and related conditions, gender identity, intersex status, and sexual orientation;
- “Significant update” means a new version, new release, or other update to an automated decision tool that materially changes its principal use, principal intended use, or outcome.
- Requires a deployer to perform an impact assessment on any automated decision tool before first using it and annually thereafter.
- Exempts from 2) a deployer that first used an automated decision tool before the effective date of the bill.
- For a deployer who is exempt pursuant to a), instead requires them to perform an impact assessment of that automated decision tool before January 1, 2026, and annually thereafter.
- Requires a deployer to ensure that an impact assessment prepared pursuant to 2) include all of the following:
- A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts;
- A description of the automated decision tool’s outputs and how they are used to make, or be a substantial factor in making, a consequential decision;
- A summary of the categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
- Each category of personal information identified by reference to the applicable subparagraph enumerated under the specified subparagraph of the California Consumer Privacy Act;
- Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under the specified subparagraph California Consumer Privacy Act;
- Each category of information related to a natural person’s receipt of sensitive services, as defined in the Confidentiality of Medical Information Act, identified by reference to the specific category of sensitive service enumerated in the definition;
- A statement of the extent to which the deployer’s use of the automated decision tool is consistent with or varies from the statement required by 12);
- An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployer’s use of the automated decision tool;
- A description of the safeguards implemented, or that will be implemented, by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the automated decision tool known to the deployer at the time of the impact assessment;
- A description of how the automated decision tool will be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision;
- A description of how the automated decision tool has been or will be evaluated for validity or relevance.
- Requires a developer, before making an ADT that it designs, codes, or produces available to potential deployers, to perform an impact assessment on the ADT and annually thereafter.
- Exempts from 4) a developer that first made an ADT available to potential deployers before the effective date of the bill.
- Requires a developer exempt pursuant to a), to instead perform an impact assessment on the ADT before January 1, 2026, and annually thereafter.
- Requires a developer to ensure that an impact assessment prepared pursuant to 4) includes all of the following:
- A statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts;
- A description of the ADT’s outputs and how they are used to make, or be a substantial factor in making, a consequential decision;
- A summary of the categories of information collected from natural persons and processed by the ADT when it is used to make, or be a substantial factor in making, a consequential decision, including, but not limited to, all of the following:
- Each category of personal information identified by reference to the applicable subparagraph enumerated under the specified subparagraph of the California Consumer Privacy Act;
- Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under the specified subparagraph California Consumer Privacy Act.
- Each category of information related to a natural person’s receipt of sensitive services, as defined in the Confidentiality of Medical Information Act, identified by reference to the specific category of sensitive service enumerated in the definition.
- An analysis of a potential adverse impact on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information from the deployer’s use of the ADT;
- A description of the measures taken by the developer to mitigate the risk known to the developer of algorithmic discrimination arising from the use of the ADT;
- A description of how the ADT can be used by a natural person, or monitored when it is used, to make, or be a substantial factor in making, a consequential decision.
- Requires a deployer or developer, in addition to the impact assessment required by 2) and 4), to perform, as soon as feasible, an impact assessment with respect to any significant update.
- Exempts from 2) – 6) deployers with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an ADT that impacted more than 999 people per year.
- Requires a deployer, at or before the time an ADT is used to make a consequential decision, to notify any natural person that is the subject of the consequential decision that an ADT is being used.
- Requires a deployer to provide to a natural person notified pursuant to 8) all of the following:
- A statement of the purpose of the ADT;
- Contact information for the deployer;
- A plain language description of how the ADT that includes a description of any human components and how any automated component is used to inform a consequential decision;
- Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the ADT, as provided in 10) – 11).
- Requires a deployer to, if a consequential decision is made solely based on the output of an ADT, and if technically feasible, accommodate a natural person’s request to not be subject to the ADT and to be subject to an alternative selection process or accommodation.
- Authorizes a deployer, after a request pursuant to 10), to reasonably request, collect, and process information from a natural person for the purposes of identifying the person and the associated consequential decision. If the person does not provide that information, the deployer is not obligated to provide an alternative selection process or accommodation.
- Requires a developer to provide a deployer with a statement regarding the intended uses of the ADT and documentation regarding all of the following:
- The known limitations of the ADT, including any reasonably foreseeable risks of algorithmic discrimination arising from its intended use;
- A description of the type of data used to program or train the ADT;
- A description of how the ADT was evaluated for validity and explainability before sale or licensing;
- A description of the deployers responsibilities pursuant to this bill.
- Exempts the disclosure of trade secrets, as defined by the referenced section of the Civil Code.
- Requires a deployer or developer to establish, document, implement, and maintain a governance program that contains reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use or intended use of an ADT.
- Requires the safeguards required by 14) to be appropriate to all of the following:
- The use or intended use of the ADT;
- The deployer’s or developer’s role as a deployer or developer;
- The size, complexity, and resources of the deployer or developer;
- The nature, context, and scope of the activities of the deployer or developer in connection with the ADT;
- The technical feasibility and cost of available tools, assessments, and other means used by a deployer or developer to map, measure, manage, and govern the risks associated with an ADT.
- Requires the governance program required by 14) to be designed to do all of the following:
- Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this bill;
- Requires an employee designated pursuant to a) to have the authority to assert to the employee’s employer a good faith belief that the design, production, or use of an ADT fails to comply with the requirements of this bill;
- Requires an employer of an employee designated pursuant to a) to conduct a prompt and complete assessment of any compliance issue raised by that employee.
- Identify and implement safeguards to address reasonably foreseeable risks of algorithmic discrimination resulting from the use or intended use of an ADT;
- If established by a deployer, provide for the performance of impact assessments as required by 2) and 4);
- If established by a developer, provide for compliance with 8) and 12);
- Conduct an annual and comprehensive review of policies, practices, and procedures to ensure compliance with this bill;
- Maintain for five years after completion the results of an impact assessment;
- Evaluate and make reasonable adjustments to administrative and technical safeguards in light of material changes in technology, the risks associated with the ADT, the state of technical standards, and changes in business arrangements or operations of the deployer or developer.
- Designate at least one employee to be responsible for overseeing and maintaining the governance program and compliance with this bill;
- Exempts from the requirements of 14) – 16) a deployer with fewer than 25 employees unless, as of the end of the prior calendar year, the deployer deployed an ADT that impacted more than 999 people per year.
- Requires a deployer or developer to make publicly available, in a readily accessible manner, a clear policy that provides a summary of both of the following:
- The types of ADT currently in use or made available to others by the deployer or developer;
- How the deployer or developer manages the reasonably foreseeable risks of algorithmic discrimination that may arise from the use of the ADT it currently uses or makes available to others.
- Prohibits a deployer from using an ADT that results in algorithmic discrimination.
- Prohibits a developer from making available to potential deployers an ADT that results in algorithmic discrimination.
- Authorizes the Civil Rights Department (CRD) to investigate a report of algorithmic discrimination or any other violation of the requirements imposed by this bill.
- Upon the request of the CRD, requires a deployer or developer to, within seven days of the request, provide any impact assessment that it performed to the CRD.
- Establishes that disclosure of an impact assessment pursuant to 22) does not constitute a waiver of any attorney-client privilege or work-product protection that might otherwise exist with respect to the impact assessment and any information contained in the impact assessment.
- Exempts a trade secret, as defined, contained in an impact assessment disclosed to the CRD pursuant to 22) from the California Public Records Act (CPRA).
- Requires the CRD or a public prosecutor with which an impact assessment was shared pursuant to 28), when complying with a request pursuant to the CPRA, to redact any trade secret from the impact assessment.
- Makes a deployer or developer who violates 22) liable for an administrative fine of not more than $10,000 per violation in an administrative enforcement action brought by the CRD.
- Makes each day on which an ADT is used for which an impact assessment has not been submitted pursuant to 22) a distinct violation.
- Authorizes the CRD to provide an impact assessment it receives to a public prosecutor listed in 30) to assist that public entity in initiating or litigating a civil action under that section.
- Authorizes the CRD to develop rules or regulations necessary to effectuate the intent of 21) – 28).
- Authorizes any of the follow public entities to bring a civil action against a deployer or developer for a violation of this bill:
- The Attorney General in the name of the people of the State of California;
- A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred;
- A city prosecutor in any city having a full-time prosecutor, with the consent of the district attorney;
- The Civil Rights Department.
- Authorizes a court to award all of the following in an action brought pursuant to 30);
- Injunctive relief;
- Declaratory relief;
- Reasonable attorney’s fees and litigation costs;
- Only in an action for a violation involving algorithmic discrimination, a civil penalty of $25,000 per violation.
- Requires a public attorney or the Civil Rights Department, before commencing an action for injunctive relief, to provide 45 days’ written notice to a deployer or developer of the alleged violations.
- Permits the developer or deployer to cure, within 45 days of receiving the written notice described in 32), the noticed violation and provide the person who gave the notice an express written statement, made under penalty of perjury, that the violation has been cured. If the developer or deployer cures the noticed violation and provides the express written statement, a claim for injunctive relief shall not be maintained for the noticed violation.
- Makes it unlawful for a deployer or developer to retaliate against a natural person for that person’s exercise of rights provided for under this chapter.
- Exempts cybersecurity-related technology from the bill’s provisions.
- Establishes that the rights, remedies, and penalties established by this bill are cumulative and cannot be construed to supersede the rights, remedies, or penalties established under other laws including, but not limited to the Fair Employment and Housing Act (FEHA) and the Unruh Civil Rights Act.
EXISTING LAW:
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Government Code Section 12930.)
- Enacts the Fair Employment and Housing Act. (Government Code Sections 12900 et seq.)
- Enacts the Unruh Civil Rights Act. (Civil Code Section 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
- Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
- Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civil Code Section 3426.1 (d).)
- Enacts the California Consumer Privacy Act. (Civil Code Section 1798.100 et seq.)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS: For years, automated decision tools (ADT) (also known as automated systems, or automated decision technology) have been integrated into our lives. Briefly defined, ADT are technological tools that make individualized decisions based on a coding structure. While the code and development of the automated technology is manmade, the ultimate decision is theoretically void of any human input. Most of us take automated technology and its algorithms for granted, often never recognizing when they are impacting our day-to-day lives. While this ensures that we can move about our lives unimpeded, it also means that we are often unaware of the prevalence of this relatively new technology. Like most any new development, automated technology has both positive and negative ramifications.
We are now faced with the question of how to apply existing federal and state legal protections to the world of ADT. In theory, ADT is a technology immune from the misperceptions and human prejudices. Unfortunately, due to its development by humans, ADT is often imbued with inherent bias. These biases include preconceptions based on race, gender, sex, geographic origin, and any other number of characteristics that humans themselves have developed over generations. According to the author:
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADT’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
This bill takes inspiration from the 2023 White House’s AI Bill of Rights to ensure ADT is developed and deployed in a responsible manner. In October 2022, the White House released a white paper detailing policy proposals on how to ensure that artificial intelligence (AI), including ADT, works best for everyone. Titled The Blueprint for an AI Bill of Rights, the paper identifies five main discussion points: safe and effective systems; algorithmic discrimination protections; data privacy; notice and explanation; and human alternatives, consideration, and feedback. (The Blueprint for an AI Bill of Rights, The White House (October 2022) available at https://www.whitehouse.gov/ostp/ai-bill-of-rights/.) Recognizing the breadth of potential technology that may be captured in the discussion of the AI Bill of Rights, the white paper applies a two-part test to narrow its scope to: 1) automated systems; that 2) have the potential to meaningfully impact the American public’s rights, opportunities, or access to critical resources or services. (Id. at p. 8.) The paper further identifies rights, opportunities, or access to mean 1) civil rights, civil liberties, and privacy; 2) equal opportunities; and 3) access to critical resources or services. (Ibid.) The White House’s AI Bill of Rights also includes definitions for numerous phrases and concepts implicated by the paper, including algorithmic discrimination and automated system. (Id. at p.10.) The AI Bill of Rights then goes on to lay out expectations for each of the five principles, and what should be expected by the public with regard to each principle. For example, in its discussion of automated systems, the paper posits that “[t]he public should be consulted in the design, implementation, deployment, acquisition and maintenance phases of automated system development,” and that:
[B]efore deployment, and in a proactive and ongoing manner, potential risks of the automated system should be identified and mitigated. Identified risks should focus on the potential for meaningful impact on people’s rights, opportunities, or access and include those to impacted communities that may not be direct users of the automated system, risks resulting from purposeful misuse of the system, and other concerns identified via the consultation process. (Id. at p.18.)
The paper also suggests that:
[A]utomated systems should have ongoing monitoring procedures, including recalibration procedures, in place to ensure that their performance does not fall below an acceptable level over time, based on changing real-world conditions or deployment contexts […] and should include continuous evaluation of performance metrics and harm assessments, updates of any system, and retraining of any machine learning models as necessary[.] (Id. at p.19)
Jumping to the AI Bill of Rights’ section on “Notice and Explanation,” the paper states: “You should know that an automated system is being used, and understand how and why it contributes to outcomes that impact you.” (Id. at p. 40.)
It appears that this bill is largely modeled on the White House’s AI Bill of Rights, as evidenced by the various reporting and notice requirements included in its language, discussed in further detail below.
This bill proposes a new framework to regulate the development and deployment of ADTs. The bill would impose numerous requirements on both developers (those who build ADT) and deployers (those who implement ADT) of ADT. This bill is substantially similar to AB 331 (Bauer-Kahan, 2023), with a number of key differences.
Definitions. The bill provides key definitions that establish the foundation of the bill; two of which are discussed in detail below.
First, the bill defines “algorithmic discrimination” as “the condition in which an [ADT] contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.” A number of organizations opposed to the measure, including the California Chamber of Commerce, identify this definition as a source of concern. They argue that “algorithmic discrimination, or discrimination that results from AI, is already addressed by our anti-discrimination laws which are rights based and not technology specific,” and point to the Fair Employment and Housing Act (FEHA) or the Unruh Civil Rights Act as examples.
California’s Unruh Civil Rights Act and FEHA prohibit discrimination on the basis of enumerated “protected characteristics.” While the lists of protected characteristics are not identical, they are quite similar. They both include, among other things, “sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status.” (See e.g. Civil Code Section 51.) Provisions of the Government Code, which includes FEHA, provide a similar list of protected characteristics in different sections prohibiting discrimination in employment, housing, and participation in state-funded programs. (See Government Code Sections 11135, 12940, and 12955.) However, FEHA and the Unruh Act are by no means the only provisions of existing law that prohibit discrimination on the basis of enumerated characteristics. For example, provisions of the Education Code prohibit discrimination in any program or activity conducted by an educational institution that receives, or benefits from, state financial assistance, or enrolls pupils who receive state student financial aid. (See especially Education Code Section 200 et seq.)
Simply tying the definition of “algorithmic discrimination” to either FEHA or Unruh, while arguably straightforward, may pose difficulties considering the scope of this bill. While FEHA and Unruh target discrimination in some settings (employment, housing, and discrimination by private businesses), this bill targets discrimination in the same categories, as well as an extensive list of services and industries such as education, utilities, family planning, adoption services, health care or health insurance, financial services, criminal justice, legal services, and voting. Some of these categories are likely covered by the Unruh Act, such as health insurance and financial services. However others, such as education, utilities, or health care, may not be.
Nonetheless, the opponents point is worth considering. Ultimately, the bill appears to prohibit discrimination that happens to have occurred as the result of use of an ADT. Arguably, discrimination in all the industries listed in the bill are covered, in one way or another, by our state’s extensive anti-discrimination statutes.
It should be noted that the most recent set of amendments also includes a provision clarifying that the rights proffered by the bill are in addition to, rather than supplanting, existing legal protections against discrimination. The language at proposed Section 22756.11 specifically identifies that the bill’s protections are cumulative of the rights established by both FEHA and the Unruh Civil Rights Act.
Second, the bill defines “automated decision tool” (ADT) to mean “a system or service that uses artificial intelligence (AI) and has been specifically developed to, or specifically modified to, make, or be a substantial factor in making, consequential decisions.” In essence, an ADT is 1) a system or service; that 2) uses AI; and 3) has been specifically developed or modified; to 4) make or be a substantial factor in making a consequential decision. The most recent amendments to the bill changed a portion of this definition to require the ADT to be a “substantial factor,” rather than “controlling factor,” in making the resulting decision. (This is also a change from last year’s AB 331, which also used the term “controlling factor.”)
ADT’s role in decision making: “controlling” versus “substantial” factor. Letters submitted prior to this amendment expressed concern that the term “controlling factor” was both too expansive and not expansive enough. For example, the Chamber of Commerce and their coalition argued, “it captures even those systems where human judgment and supervision is involved and are ultimately the controlling factors in making the decision – not the ADT itself.” On the other hand, a coalition of organizations in a support if amended position, including Equal Rights Advocates and the Service Employees International Union (SEIU), requested a change to “controlling factor” to “prevent deployers from avoiding compliance with the bill simply by having a human rubber stamp algorithmic recommendations.” Neither of these positions have been updated since the bill was amended in the Assembly Privacy and Consumer Protection Cmte.
The “substantial factor” test is used in tort law to establish whether certain conduct was the proximate cause of the alleged injury. It can be implemented instead of the “but-for” test and allows greater opportunity to argue that an alleged injury was actually caused by a different, more influential factor. Proximate cause and substantial factor determinations present similar analyses to the question posed by the bill of whether an incident of discrimination was the result of an ADT or independent human activity.
Arguably, “substantial factor” is a less exacting standard than “controlling factor” and would alleviate the concern identified by the coalition in the support if amended position. In the event a human is simply accepting the results provided by an ADT without exerting any additional influence themselves, it would be difficult to discern how the ADT would not have been a “substantial factor” in any resulting discrimination. To the extent the author wishes to clarify any remaining ambiguity on the question of what constitutes a “substantial factor,” the analysis provided for this bill from the prior Committee captures the issue quite eloquently:
A number of stakeholders have expressed concerns that “controlling” sets too high of a bar, excluding circumstances in which an ADT may have influenced but not decided the outcome. On the other hand, the author does not want to capture circumstances in which an algorithm plays a minor role that is incapable of altering the outcome of the decision[.]
It seems that a potential definition for the term “substantial factor” could be developed from this language to mean instances when the algorithm is capable of altering the outcome of the decision.
Obligations of developers and deployers under the bill. The thrust of this bill focuses on imposing numerous obligations on developers and deployers of ADT. Developers refer to entities that build and provide ADTs, and deployers refer to entities that use ADTs. It’s important to note that a developer could also be a deployer of an ADT.
The bill sets requirements for both deployers and developers of ADT to perform impact assessments, meaning a documented risk-based evaluation, of an ADT that meets specified criteria, at certain intervals. Proposed Section 22756.1 requires deployers to perform an impact assessment on an ADT prior to its first use and annually thereafter. For deployers that have already used an ADT before the effective date of the bill, the requirement is slightly modified to perform an impact assessment by January 1, 2026 and then for each subsequent year. The bill imposes similarly timed requirements for developers, except that the impact assessment must be performed before making the ADT available to potential deployers, as well as for each significant update. Each impact assessment is required to include specified information including the ADT’s intended benefits, uses, and deployment contexts, and an analysis of potential adverse impacts on the basis of specified protected characteristics.
AB 331 also required deployers and developers to complete annual impact assessments; but it required them to be submitted annually to the Civil Rights Department (CRD). This bill takes a different approach. Deployers and developers are not required to submit their annual impact assessments to CRD. Instead the department is authorized to request an annual impact assessment should a complaint be filed against the business. (This authorization is discussed in further detail in the enforcement portion of this analysis, below.)
Under the bill, deployers have an additional notice requirement that is not imposed on developers. At or before the time the ADT is used to make a consequential decision, a deployer is required to notify a person that the ADT is being used. The notice itself has various requirements including a statement of the purpose of the ADT and enough information to allow the person to request to opt-out of interacting with the ADT. Relating to this second element, deployers are also required to provide an opportunity for a person subject to an ADT to opt-out, if a consequential decision is being made solely based on the output of the ADT and if technically feasible.
Finally, both deployers and developers are required to establish a governance program structured to manage reasonably foreseeable risks related to algorithmic discrimination resulting from a use or intended use of an ADT.
Prohibition against the use and distribution of ADTs that result in algorithmic discrimination. Prior to April 22nd, the bill only prohibited an employer from using an ADT that resulted in discrimination. The most recent set of amendments, however, added a new subdivision prohibiting developers from making available to potential deployers an ADT that results in algorithmic discrimination. (Proposed Section 22756.6 (b).) This amendment seems to reinforce the underlying intent of the bill – namely to ensure, to the greatest extent possible, that ADTs that engage in algorithmic discrimination are not utilized. This amendment is also a significant change from last year’s AB 331, which only prohibited deployers from using ADTs that result in algorithmic discrimination.
Enforcement provisions. This bill proposes a relatively limited enforcement mechanism to ensure compliance, despite the bill’s vast scope. First, while last year’s version included a private right of action against deployers for use of an ADT that results in algorithmic discrimination, this bill does not. Instead, enforcement is entrusted to the CRD and public prosecutors.
The bill authorizes the CRD to investigate a report of algorithmic discrimination or any other violation of the bill’s requirements. As part of their investigative authority, the CRD can request a deployer or developer’s impact assessment. Failure to provide the assessment exposes the deployer or developer to an administrative fine of $10,000 per violation in an administrative action brought by the CRD. Notably, the CRD does not have an administrative enforcement branch. Instead, the CRD carries out its current enforcement obligations through investigation, settlement negotiations, and civil litigation. Therefore, while theoretically helpful to ensure the efficacy of the bill, authorizing the CRD to initiate an administrative action does not necessarily carry significant weight. In order to address this concern the author proposes to authorize the CRD to adopt rules and regulations as necessary. Theoretically, the CRD would be able to go through their rule-making procedures to begin the process of building out their administrative enforcement capacity.
The author’s amendment to authorize the CRD to adopt rules and regulations is as follows:
Section 22756.7 (a) The Civil Rights Department may investigate a report of algorithmic discrimination or any other violation of this chapter.
[…]
(d) The Civil Rights Department may adopt rules and regulations as necessary to effectuate the intent of this section.
Civil actions brought by CRD and public prosecutors. The bill also empowers public prosecutors as well as the CRD to initiate a civil claim against a deployer or developer for any violation of the bill’s requirements and authorizes a court to award injunctive relief, declaratory relief, reasonable attorney’s fees and costs, and, only in an action involving algorithmic discrimination, a civil penalty of $25,000 per violation. Claims only alleging a deployer or developer’s failure to complete an impact assessment, for example, would not give rise to a civil penalty.
Additionally, in claims for injunctive relief, deployers and developers are entitled to a 45-day notice and cure period. Prior to initiating the civil claim, the public prosecutor or the CRD is required to provide the developer or deployer 45 days’ notice of the alleged violation. The deployer or developer could then cure the noticed violation and provide an express written statement, made under penalty of perjury, that the violation has been cured.
Opponents argue that the bill “raises significant concerns around allowing enforcement by multiple entities and imposing punitive fines, with little to no opportunity to cure.” They specify with regard to authorization of numerous public prosecutors and, presumably, the CRD “civil enforcement should be limited to the Attorney General, particularly given the subjectivity and vagueness involved in many aspects of this bill. Otherwise, businesses can fall subject to different interpretations of the law and be found noncompliant for the same actions in one jurisdiction that would be compliant in another jurisdiction.”
It appears that the concern is that businesses will be subjected to a litany of litigation from all of the public prosecutors, on top of the CRD’s administrative enforcement and new litigation authority. However, the concern fails to consider that any litigation initiated by any of the authorized prosecutors would likely rely on an analysis of the impact assessments and subsequent investigation conducted by the CRD. Presumably, the CRD would track and manage requests for impact assessments and potential litigation coming from public prosecutors. It is unlikely that the enforcement provisions will present public prosecutors with an opportunity to use their already limited resources to file meritless litigation.
Fines. The opposition also cites a concern over “the punitive nature of the fines,” pointing to the potential for a $10,000 fine for a violation through an administrative action by the CRD and the $25,000 violation in a civil action for a claim of algorithmic discrimination. As noted previously, the CRD does not currently have administrative enforcement capacity and would need to build up any such department. Thus, even assuming the opposition’s contention that the fine is excessive as compared to the California Consumer Privacy Act’s fines of $2,500 per violation, it is unlikely any deployer or developer will be faced with paying an administrative penalty in the near future. Moreover, it seems relatively simple to avoid the administrative fine, even assuming the CRD’s administrative enforcement capacity, by simply providing the impact assessment upon request. Second, it is important to note that the $25,000 civil penalty is only available for claims alleging algorithmic discrimination. Thus, the vast majority of potential violations under this bill would not give rise to a $25,000 civil penalty. Instead they would only be subject to claims for injunctive or declaratory relief, and reasonable attorney’s fees and costs.
Notwithstanding any consideration of the likelihood of the available penalties review of the proposed penalties would likely survive scrutiny under the Eighth Amendment’s prohibition on unreasonable fines. The prohibition against unreasonable fines has been applied to government imposed penalties for over two decades. (Hudson v. United States (1997) 522 U.S. 93.) To determine if either penalty is reasonable, or unlawfully excessive, the case law in this area generally requires a court to look at the proportionality of the penalty amount in comparison to the underlying violation. (United States v. Bajakajian (1998) 524 U.S. 321.) Instances of algorithmic discrimination in the number of industries identified by this bill could severely impact the individual bringing the claim. For example, an ADT developed to review applications for rental units that engages in algorithmic discrimination could result in individuals being unable to obtain safe housing. An ADT developed to identify health risks that engages in algorithmic discrimination could result in alerting a patient to a condition that does not exist or, even more dangerously, could fail to suggest the proper course of treatment. Thus, it could be reasonably argued that even the heftier fine of $25,000 is not disproportionate to a substantiated claim of algorithmic discrimination.
Right to Cure. Finally, the opposition makes clear their appreciation for the bill’s right to cure, but posits that the cure provision “is inadequate and illusory.” They argue that the bill “requires [the developer or deployer] to not only cure the noticed violation, but also provide the person giving the notice an express written statement that the violation has been cured and that no further violations shall occur. Due to the complexity and evolving nature of this technology, as well as the bill’s breadth and vagueness issues, it is unrealistic for [a] company to be asked to sign such a statement.”
Proposed Section 22756.8 (b)(2)(A) provides a developer or deployer a right to cure within 45 days of receiving written notice, and requires only a statement, made under penalty of perjury, that the violation has been cured. The subparagraph includes no reference to an assurance that no further violations will occur. This requirement was present in last year’s AB 331, however, the current measure does not require such an attestation from a developer or deployer. Under this bill’s language, it seems clear that a developer or deployer faced with a notice of pending litigation could simply cease using an ADT or pull the product from the market, at which point the prosecutor would no longer be authorized to seek injunctive relief. In sum, despite concerns to the contrary, it appears that the enforcement provisions in this bill are relatively narrow and likely to give rise to potential liability only in the most egregious circumstances.
Additional author’s amendments. The author also proposes various technical amendments to correct cross-references and update references to reflect the CRD’s authorization to initiate civil litigation. They are as follows:
Section 22756.8
(b)(1) A public attorney or the Civil Rights Department, before commencing an action pursuant to this section for injunctive relief, shall provide 45 days’ written notice to a deployer or developer of the alleged violations of this chapter.
[…]
(b)(3)(B) In complying with a request pursuant to the California Public Records Act (Division 10 (commencing with Section 7920.000) of Title 1 of the Government Code), the Civil Rights Department, or an entity a public prosecutor with which whom an impact assessment was shared pursuant to subdivision (d), (c), shall redact any trade secret, as defined in Section 3426.1 of the Civil Code, from the impact assessment.
ARGUMENTS IN SUPPORT: This bill is supported by Secure Justice. They submit:
Secure Justice is a non-profit organization located in the San Francisco Bay Area, California, that advocates against state abuse of power, and for reduction in government and corporate overreach.
We target change in government policy and contracting, and corporate complicity with government policies and practices that are inconsistent with democratic values and principles of human rights.
We believe AB 2930 is a critical need in this fast-emerging world of AI decision making. Sufficient research exists to demonstrate that many individuals are being actively harmed today as AI proceeds without sufficient oversight, transparency into the decision making, and lack of accountability when discriminatory decisions occur. AB 2930 addresses each of these concerns.
For these reasons, AB 2930 would further Secure Justice’s goals of guarding against erosion of our civil liberties and right to privacy. We are proud to support AB 2930 and encourage your “aye” vote when it is heard in your committee.
A coalition of workers rights advocates, labor unions, and social justice organizations have submitted a position of support if amended. They write the following:
California has the opportunity to lead the nation with innovative policy that places common-sense guardrails on the development and use of AI and automated decision-making tools. Legislating on emerging technology requires thoughtful and deliberative work, and engagement with all the stakeholders who will be impacted. We commend you for taking on this challenge.
We would like to work with you to ensure this bill provides needed protections for the people of California. In its current form, however, we are concerned that it would not provide necessary transparency measures, that some of the definitions would allow companies to side-step the critical accountability mechanisms this bill creates, and that consumers and job candidates would not receive sufficient notice or explanation of consequential decisions made with the help of automated decision tools, among other concerns. Furthermore, when the bill was reintroduced in 2024 as AB 2930, enforcement provisions were weakened and overbroad protections for “trade secrets” were added.
The most important changes we think should be made include:
1. Revising the bill’s definitions (and its “controlling factor” requirement, in particular) to prevent deployers from avoiding compliance with the bill simply by having a human rubber stamp algorithmic recommendations;
2. Excluding worker management and other issues related to ongoing employment because of the diverse sets of laws that apply, the unique power dynamics, and the wide range of harms that go beyond discrimination that can occur when AI is deployed in the workplace;
3. Strengthening the notice and explanation provisions so that workers and consumers have a meaningful understanding of how and why they are assessed by AI systems;
4. Strengthening the impact assessment provisions by requiring unbiased, independent auditors and that all forms discrimination must be assessed and mitigated;
5. Ensuring the bill has adequate enforcement remedies and does not undermine existing laws.
ARGUMENTS IN OPPOSITION: This bill is opposed by a coalition of business advocates including the California Chamber of Commerce, TechNet, and the California Retailers Association. It is also independently opposed by the Electronic Frontier Foundation (EFF). EFF submits:
One concern lies in the key definition of "automated decision tool," on which most of the bill’s force turns. But the bill defines an automated decision tool narrowly as one that “has been specifically developed and marketed to, or specifically modified to, make, or be a controlling factor in making, consequential decisions.” Leaving aside our distaste for the “controlling factor” rubric, this language may enable entities to evade regulation entirely.
Nothing in the bill indicates how one would verify a deployer or developer's assertion about whether they use or designed a system to be a "controlling factor" in such decisions—or that anyone would. Indeed, current text requires that they “specifically” do so, and unclearly also seems to require specific market[ing]” about being a “controlling factor.” Essentially, this allows entities to self-certify their way out of regulation, and leaves those affected by these decisions with no recourse to challenge outputs or even know that they are subject to such a tool. This loophole should be closed.
Second, A.B. 2930, in covering private and government entities and several areas of consequential decisions, seeks to offer a one-size-fits all solution to an issue that is too complex for a single set of standards. While there may be general common principles for evaluating automated decisionmaking tools, implementing such standards depends heavily on context.
Specifically, we have concerns that A.B. 2930 conflicts or unclearly interacts with established California law. For example, the bill defines "algorithmic discrimination" using the undefined phrase, "unjustified differential treatment of impacts disfavoring people based on" a variety of protected characteristics. The long-standing Unruh Act more broadly asserts that "All persons within the jurisdiction of this state are free and equal, and [...]are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever." Existing law also lays out specific circumstances that are or are not considered discrimination in relation to housing, senior services, services for disabled people, and more. Adopting this new definition of discrimination without acknowledging or addressing existing antidiscrimination laws could weaken current protections against discrimination if a company or the government uses an algorithm—and will invite expensive litigation.
Decisions made by an algorithm or automated decisionmaking tool deserve no unique treatment in terms of discrimination, nor do we believe that is the intent of the bill. However, if the bill does not reckon with these interactions, it will be left for courts to wrestle for years to come. To offer another example, we would like clarity on how A.B. 2390 interacts with Labor Code § 2100 (warehouse distribution centers). Because A.B. 2930 covers "automated task allocation" but does not contain the specific guardrails around the use of "employee work speed data" specified there, it also leaves compliance as an open question.
Such confusion does not benefit workers, consumers, or the state of California. A.B. 2930 has the opportunity to set a strong path not only for this state, but for others across the country. It should be crafted very carefully.
We encourage the author to continue to engage and expand discussions on the effects of this bill, especially with those most directly affected by automated decisionmaking such as civil rights lawyers, workers, specialized advocates who work in the named areas specified in the bill, and consumers. We also encourage considering bills with a smaller scope to allow for deep investigation into possible interactions. Policy will be stronger, better, and more effective with more feedback and time.
Internet.Works has assumed a position of oppose unless amended and submits:
We are concerned that such a heavy-handed early intervention would have a chilling effect on the use of algorithms in decision-making and the competition for underling research and development. While there have been ugly examples of this technology contributing to bias and discrimination, there are many more examples of it being used to promote fairness and equity, taking out many of the subjective factors that contribute to systemic inequities. For smaller industry players represented by this coalition, such onerous impact assessment and consumer notice requirements that are burdensome and anticompetitive. Most importantly, we emphasize the need to consider a risk-based assessment of functions and decision types. While the bill attempts to do this with the enumerated list of “consequential decisions,'' it needs additional layers of specificity. Not addressing this nearly unlimited regulatory burden would put less resourced platforms at a severe competitive disadvantage as the utilization of ADT technology would come with a heavy up front cost, regardless of intended or actual use.
To accomplish this we suggest the following minor amendments to the operative definitions in the beginning of the bill:
(a) “Algorithmic discrimination” means the condition in which an automated decision tool creates a legally prohibited contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
(b) “Artificial intelligence” means a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing a real or virtual environment.
(c) “Automated decision tool” means a system or service that uses artificial intelligence and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling factor in making, consequential decisions.
(d) “Controlling Factor” means a factor that is weighted more than all other factors combined.
REGISTERED SUPPORT / OPPOSITION:
Support
Secure Justice
Support if amended
American Federation of Musicians, Local 7
BSA the Software Alliance
California Employment Lawyers Association
California Nurses Association
Center for Democracy and Technology
Center on Race and Digital Justice
Consumer Reports
East Bay Community Law Center
Economic Security California Action
Equal Rights Advocates
Greenlining Institute
Legal Aid At Work
Oakland Privacy
Rise Economy
SEIU California
Techequity Action
Opposition
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
Association of California Life and Health Insurance Companies
Association of National Advertisers
California Association of Realtors
California Bankers Association
California Chamber of Commerce
California Community Banking Network
California Credit Union League
California Financial Services Association
California Hotel & Lodging Association
California Manufacturers & Technology Association
California Mortgage Bankers Association
California Retailers Association
California Staffing and Recruiting Association
Chamber of Progress
Civil Justice Association of California
Computer and Communications Industry Association
CTIA
Electronic Frontier Foundation
Insights Association
Los Angeles Area Chamber of Commerce
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Personal Insurance Federation of California
State Privacy and Security Coalition, INC.
TechNet
Oppose unless amended
Internet.works
Other
Alliance for Digital Innovation
California Radiological Society
Analysis Prepared by: Manuela Boucher-de la Cadena / JUD. / (916) 319-2334
Date of Hearing: April 16, 2024
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
ABPCA Bill Id:AB 2930 (
Author:Bauer-Kahan) – As Introduced Ver:February 15, 2024
AS PROPOSED TO BE AMENDED
SUBJECT: Automated decision tools
SYNOPSIS
This author-sponsored measure is a significant, meaningful attempt to set state policy with respect to private and public use of automated decision tools (ADTs): systems and services that employ artificial intelligence, and make, or are a controlling factor in making, consequential decisions. Consequential decisions, in turn, are those that materially impact the necessities of a person’s life, such as housing, education, employment, and health care (including reproductive health care), as well as the person’s involvement with critical public functions such as voting and the criminal justice system.
The bill would establish requirements for informed, conscious deployment of ADTs, based on the White House’s recently-released Blueprint for an AI Bill of Rights. Broadly speaking, the bill only prescribes one policy outcome: it forbids algorithmic discrimination against individuals on the basis of characteristics protected under state civil rights law, such as race, age, national origin, disability, limited English proficiency, and sex (including pregnancy, childbirth, and related conditions; gender identity; intersex status; and sexual orientation). Much of the bill is instead devoted to ensuring that ADT developers and deployers thoroughly consider the design, testing, and functioning of ADTs before releasing them into the world. This appears to be sound policy, as developers and deployers are the entities that possess the most knowledge about the systems they work with.
This bill is supported by the Microsoft Corporation and Workday. It is opposed by a variety of business trade associations, including the California Chamber of Commerce. A broad coalition of organizations, including Tech Equity, Oakland Privacy, and SEIU California, takes a support-if-amended position. Author’s amendments respond to some of their requested amendments and make other clarifying changes.
If passed by this Committee, this bill will next be heard by the Assembly Judiciary Committee.
SUMMARY: Establishes a statutory framework to further the safe and informed use of ADTs in California. Specifically, this bill:
- Requires that developers and deployers of ADTs perform impact assessments on or before January 1, 2026, and annually thereafter. Exempts deployers with fewer than 25 employees unless they deploy ADTs that impact more than 999 people.
- Requires that impact assessments include:
- A statement of the purpose of the ADT and its intended benefits and uses.
- A description of the ADT’s outputs and how they are used to make consequential decisions, or be a controlling factor in making those decisions.
- A summary of types of data collected from natural persons and processed by the ADT.
- A statement of how a deployer’s use of an ADT differs from the ADT purpose as stated by the developer.
- An analysis of potential adverse impacts on the basis of sex, race, color, ethnicity, religion, age, national origin, limited English proficiency, disability, veteran status, or genetic information resulting from a deployer’s use of the ADT.
- A description of safeguards implemented by the deployer to address any reasonably foreseeable risks of algorithmic discrimination arising from the use of the ADT.
- A description of how the ADT will be used by a natural person, or monitored when it is used, to make consequential decisions.
- Requires developers and deployers to perform impact assessments if an ADT they develop or deploy receives a significant update.
- Requires that deployers notify individuals that are subject to a consequential decision made by an ADT, and that they provide subject individuals with relevant information.
- Requires deployers to accommodate an individual’s request to not be subject to an ADT’s decision, if technically feasible, and requires the deployer to provide an alternative process. Permits the deployer to capture relevant information from the individual upon accommodating their request.
- Requires a developer to provide a deployer with a statement regarding the intended uses of an ADT they provide, along with documentation relating to the ADT’s known limitations, training data, and how the tool was evaluated for validity and explainability. Does not require disclosure of trade secrets.
- Requires developers and deployers to establish governance programs, maintained by at least one employee, that assess and address risks of algorithmic discrimination arising from the use of ADTs.
- Requires developers and deployers to maintain impact assessments performed on ADTs for at least two years.
- Requires developers and deployers to make publicly available documentation related to the types of ADTs currently used or made available by the entity, as well as how the entity manages algorithmic discrimination risks arising from the use of these ADTs.
- Prohibits deployers from using discriminatory ADTs.
- Permits the Civil Rights Department (CRD) to investigate reports of algorithmic discrimination. Requires developers and deployers to provide impact assessments to the CRD upon request.
- Permits the CRD to bring an administrative enforcement action not exceeding $10,000 per violation against any developer or deployer who violates the requirement to disclose impact assessments.
- Permits the Attorney General, district attorneys, a county counsel, city attorneys, and city prosecutors to bring a civil action against a developer or deployer for a violation of the bill. Permits a court to award injunctive relief, declaratory relief, reasonable attorney’s fees and litigation costs, and, in cases where a violation involves algorithmic discrimination, a civil penalty of $25,000 per violation.
- Requires a developer or deployer to be provided with 45 days written notice before an action is commenced against them, during which they can cure the noticed violation.
- Exempts cybersecurity-related technologies from its provisions.
- Defines various terms, including:
- Algorithmic discrimination: the condition in which an ADT contributes to unjustified differential treatment or impacts disfavoring people based on their actual or perceived race, color, ethnicity, sex, religion, age, national origin, limited English proficiency, disability, veteran status, genetic information, reproductive health, or any other classification protected by state law.
- Artificial intelligence: an engineered or machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing a real or virtual environment.
- ADT: a system or service that uses artificial intelligence and has been specifically developed and marketed to, make, or be a controlling factor in making, consequential decisions.
- Consequential decision: a decision or judgment that has a legal, material, or similarly significant effect on an individual’s life relating to access to government benefits or services, assignments of penalties by government, or the impact of, or the cost, terms, or availability of several listed categories.
- Deployer: a person, partnership, state or local government agency, or corporation that uses an ADT to make a consequential decision.
- Developer: a person, partnership, state or local government agency, or corporation that designs, codes, or produces an ADT, or substantially modifies an artificial intelligence system or service for the intended purpose of making, or being a controlling factor in making, consequential decisions, whether for its own use or for use by a third party.
- Impact assessment: a documented risk-based evaluation of an ADT.
- Significant update: a new version, new release, or other update to an ADT that materially changes its principal use, principal intended use, or outcome.
EXISTING LAW:
- Establishes the Civil Rights Department, and sets forth its statutory functions, duties, and powers. (Gov. Code § 12930.)
- Establishes the Fair Employment and Housing Act. (Gov. Code § 12900 et seq.)
- Establishes the Unruh Civil Rights Act. (Civil Code § 51.)
- Defines “trade secret” under the Uniform Trade Secrets Act as information, including a formula, pattern, compilation, program, device, method, technique, or process, that both:
- Derives independent economic value, actual or potential, from not being generally known to the public or to other persons who can obtain economic value from its disclosure or use; and
- Is the subject of efforts that are reasonable under the circumstances to maintain its secrecy. (Civ. Code § 3426.1(d).)
- Establishes the California Privacy Protection Agency (Privacy Agency) and vests it with full administrative power, authority, and jurisdiction to implement and enforce the California Consumer Privacy Act (CPPA) of 2018. (Civ. Code § 1798.199.10.)
- Requires the Attorney General to issue regulations governing access and opt-out rights with respect to businesses’ use of automated decisionmaking technology. Passes along this authority to the California Privacy Protection Agency after a set period of time. (Civ. Code § 1798.185.)
FISCAL EFFECT: As currently in print this bill is keyed fiscal.
COMMENTS:
- Background. Under the bill, the term “automated decision tool,” or “ADT,” refers to computer systems that are relied upon to make, or to be a controlling factor in making, consequential decisions. ADTs are used to decide whom to hire, what treatments patients should receive, whether to grant parole, and many other life-altering decisions that used to be made entirely by human beings.
Using an ADT is, in many cases, a good thing. Computers can generally draw upon far more digitally-encoded data, and process this data far more quickly, than any human being. A recent New York Times article described the development of a breast cancer screening system that uses artificial intelligence:
From the millions of cases the system is fed, the technology creates a mathematical representation of normal mammograms and those with cancers. With the ability to look at each image in a more granular way than the human eye, it then compares that baseline to find abnormalities in each mammogram.
Last year, after a test on more than 275,000 breast cancer cases, Kheiron [Medical Technologies, the system’s developer] reported that its A.I. software matched the performance of human radiologists when acting as the second reader of mammography scans. It also cut down on radiologists’ workloads by at least 30 percent because it reduced the number of X-rays they needed to read. In other results from a Hungarian clinic last year, the technology increased the cancer detection rate by 13 percent because more malignancies were identified.[1]
However, ADTs are far from flawless. Care should be taken to avoid what New York University Professor Meredith Broussard describes as “the notion that computers are more ‘objective’ or ‘unbiased’ because they distill questions and answers down to mathematical evaluation…an unwavering faith that if the world used more computers, and used them properly, social problems would disappear and we’d create a digitally enabled utopia.”[2] Many aspects of the real world are not reducible to data, and many complex situations present competing demands that human beings remain best equipped to assess.
ADT usage continues to increase. Global consulting firm McKinsey & Company reports that the adoption of artificial intelligence among the organizations it surveyed more than doubled between 2017 and 2022, with 50 percent now reporting they use the technology.[3]
ADTs use artificial intelligence. ADTs use artificial intelligence (AI) to make consequential decisions. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. This is because AI is not fundamentally different from other computer functions. Its novelty lies in its application: unlike standard computer functions, AI is able to accomplish tasks that are normally performed by humans.
Bias and discrimination can result from poor training data. There is a famous saying in computer science: “garbage in, garbage out.” The performance of an ADT is directly impacted by the quality, quantity, and relevance of the data used to train it.[4] If the data used to train the ADT is biased, the tool’s outputs will be similarly biased. Over the past thirty years, several industries have been forced to contend with this fact as they have attempted to introduce ADTs into their workflows. Specific examples follow:
Hiring and recruitment tools. It is no secret that people of various races, genders, and cultures are not distributed equally throughout the workforce. An ADT that is trained on historical data to make hiring decisions will be predisposed to maintain the ratios it is trained on; as described by Aditya Malik, the Founder and CEO of Valuematrix.ai:
Generative AI, for all its grandeur, has the potential to perpetuate latent biases inherited from human creators. A disconcerting echo of historical prejudices may inadvertently seep into the algorithms. Imagine a scenario where previous senior managers, driven by biases of gender, age, faith or race, rejected candidates for misguided reasons. The AI, if not vigilantly curated, might misconstrue these patterns as indicators of incompetence, thus exacerbating the exclusion of qualified candidates from underrepresented backgrounds.[5]
This was notoriously experienced by Amazon, who considered automating their hiring practices in the early 2010s. They opted against this approach in 2015 when they realized that their ADT-enabled system was not rating candidates in a gender-neutral way. In fact, their system was excluding women from the pool of acceptable candidates because it had been trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of inequities across the tech industry.[6]
Sentencing and bail decisions. ADTs are frequently used to inform sentencing and bail decisions. These tools are trained using historical data, and the predictions they make can therefore reflect historical bias. A 2016 Propublica study dove into the use of one such tool – COMPAS – in Broward County, Florida.[7] The study determined that Black defendants were far more likely than white defendants to be incorrectly judged to be at a higher risk of recidivism, while white defendants were more likely than Black defendants to be incorrectly flagged as low risk.
The for-profit company that developed this tool, Northpointe, does not publicly disclose the calculations used to arrive at defendants’ risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. These discrepancies mirror historical injustices perpetuated against Black Americans by California’s criminal justice system.[8]
Healthcare. When ADTs are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. In 2007, an ADT was developed to help doctors estimate whether it was safe for people who had delivered previous children through cesarean section to deliver subsequent children vaginally – a risky procedure. The ADT considered various health relevant factors as it made its decision, such as the woman’s age, her reason for the previous cesarean, and how long ago the cesarean had been performed. However, a 2017 study found that the ADT was biased; it predicted Black and Latino people were less likely to have a successful vaginal birth after a cesarean than non-Hispanic white women. As a result, doctors performed more cesareans on Black and Latino people than on white people.[9] These discrepancies perpetuate historical biases – Black Americans, for example, have historically received a lower standard of healthcare than their white counterparts.[10]
Credit and loan approval. Financial tools that utilize ADTs are similarly susceptible to bias and discrimination. An investigation by The Markup (and co-published by the Associated Press) revealed that in 2019, lenders were more likely to deny home loans to people of color; in particular, lenders were 40 percent more likely to turn down Latino applicants for loans, 50 percent more likely to deny Asian/Pacific Islander applicants, and 70 percent more likely to deny Native American applicants than similar white applicants. Lenders were 80 percent more likely to reject Black applicants than similar white applicants. In every case, the prospective borrowers of color looked almost exactly the same on paper as the white applicants, except for their race.[11]
The premises of this bill are, first, that Californians ought to be much more informed about the ADTs being used to make consequential decisions in their lives, and second, that developers and deployers of ADTs ought to be much more conscious and deliberate about what the impacts of their tools might be. In short, California ought to get a handle on ADTs while it still can before they become yet another phenomenon— like social media, greenhouse gases, guns, and urban sprawl— that could have been dealt with deliberately and intelligently in the past, but was instead allowed to expand without any constraints past the point of being manageable.
- Author’s statement. According to the author:
AB 2930 protects individuals from algorithmic discrimination by requiring developers and users to assess automated decision tools (ADTs) that make consequential decisions and mitigate any discovered biases. The use of ADT’s have become very prominent within different sectors such as housing, employment, and even in criminal justice sentencing and probation decisions. The algorithms used within ADTs can be prone to unrepresentative datasets, faulty classifications, and flawed design, which can lead to biased, discriminatory, or unfair outcomes. These tools can exacerbate the harms they are intended to address and ultimately hurt the people they are supposed to help. As the use of decision making via algorithm becomes more prevalent in our daily lives, it is crucial that we take the necessary steps to ensure that they are used ethically and responsibly.
- What this bill would do. The provisions of this bill are derived from the principles outlined in the Blueprint for an AI Bill of Rights, issued by the White House Office of Science and Technology Policy on October 4, 2022.[12] The Blueprint identifies five principles that should “guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence.” The applicable principles, which this measure generally seeks to implement, are as follows:
Safe and Effective Systems. The Blueprint summarizes this principle as meaning that “you should be protected from unsafe or ineffective systems.” The bill would implement this principle by requiring developers and deployers to complete impact assessments for any ADT that they develop or use. These impact assessments require thoughtful consideration of what ADTs are intended to do, what their effects might be, and how they are validated. Once completed, these impact assessments can be requested by the Civil Rights Department. To shield small businesses from the costs associated with this bill, deployers with fewer than 25 employees are exempted from this requirement, unless an ADT used by the deployer has impacted more than 999 people per year. Developers and deployers would have to complete new impact assessments any time a significant update, as defined, is made to an ADT.
Algorithmic Discrimination Protections. The Blueprint summarizes this principle as meaning that “you should not face discrimination by algorithms[,] and systems should be used and designed in an equitable way.” The bill would implement this principle in at least three ways:
- First, developers and deployers would be required to prepare governance programs containing reasonable administrative and technical safeguards to map, measure, manage, and govern the reasonably foreseeable risks of algorithmic discrimination associated with the use of an ADT.
- Second, at least one employee would have to be designated to oversee and maintain the governance program and compliance with this bill’s requirements. That employee would have the authority to assert to their employer a good faith belief that the design, production, or use of an ADT fails to comply with one or more requirements of the bill. The employer would then be required to conduct a prompt and complete assessment of the compliance issues raised.
- Third, the bill would flatly prohibit a deployer from using an ADT that contributes to algorithmic discrimination on the basis of any category protected under California’s civil rights laws.
Notice and Explanation. The Blueprint summarizes this principle as meaning that “you should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.” The bill would implement this principle by requiring a deployer to notify any natural person that is the subject of a consequential decision that an ADT will be used to make, or will be a controlling factor in making, that decision. The notification must be made at or before the time the ADT is used.
Human Alternatives, Consideration, and Fallback. The Blueprint summarizes this principle as meaning that “[y]ou should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.” The bill would implement this principle by requiring a deployer to accommodate a natural person’s request to not be subject to an ADT and to instead be subject to an alternative selection process or accommodation. The bill limits this requirement in two significant ways, however. First, the alternative is only available in cases in which a consequential decision is to be made solely based on the output of an ADT; i.e., one over which there is no human discretion. Second, the alternative must be technically feasible.
Ultimately, AI-powered ADTs are increasingly influencing the lives of Californians. These tools are being used to assess eligibility for loans and jobs, and influencing outcomes in healthcare and criminal justice. These tools can, when poorly designed and improperly used, undermine privacy and equity. Without a robust legal framework for their use, ADTs risk perpetuating systemic biases through opaque decision-making and a general lack of accountability. This bill seeks to tackle these risks by enforcing transparency and ensuring that decision-making tools are scrutinized and pressure-tested for discriminatory outcomes. Such an approach safeguards fundamental rights and fosters trust in the technologies that are reshaping society.
4) Comparisons to AB 331. This bill derives from AB 331, introduced in 2023 by the same author. AB 331 was passed out of two Assembly policy committees – Privacy and Consumer Protection, and Judiciary – but was ultimately held in the Assembly Appropriations Committee. Key differences between the bills include:
- First, AB 331 granted individual Californians the right to bring a civil action against a deployer who violated the bill’s provisions. The current bill does not provide for a private right of action, and gives public prosecutors civil enforcement authority.
- Second, AB 331 required developers and deployers to provide impact assessments to the Civil Rights Department within 60 days of their completion. By comparison, the current bill only requires developers and deployers to provide impact assessments to the Civil Rights Department upon request, within 7 days of the request being made.
- Third, AB 331 granted the Civil Rights Department the ability to share impact assessments with other state entities as appropriate. The current bill does not allow for sharing of impact assessments in this way. As a result, it is unclear how public prosecutors seeking civil liability are meant to support their claims of algorithmic discrimination.
5) Stakeholder positions.
In support, Microsoft Corporation writes:
AB 2930 aligns with the Microsoft Responsible AI framework and recognizes the promise and potential pitfalls that the next generation of AI may bring through the use of Automated Decision Tools (ADTs). The bill requires developers and users of ADTs to conduct and record impact assessments to evaluate the intended uses of ADTs including an analysis of potential adverse impact on the basis of race, color, ethnicity, sex, religion, age, national origin, or any other classification protected by state law. We believe AB 2930 establishes appropriate legal guardrails on the development and deployment of ADT systems through the use of impact assessments and design evaluations.
History teaches us that transformative technologies like AI require new rules of the road. Proactive, self-regulatory efforts by responsible companies will help pave the way for these new laws, but we know that not all organizations will adopt responsible practices voluntarily. In our view, effective AI regulations should center on the highest risk applications and be outcomes-focused and durable in the face of rapidly advancing technologies and changing societal expectations. To spread the benefits of AI as broadly as possible, regulatory approaches around the globe will need to be interoperable and adaptive, just like AI itself. We believe AB 2930 moves California in the right direction to ensure that AI is built and used responsibly and ethically.
A coalition of opponents from business trade associations sets forth what it sees as the key problems with the bill:
- Scoping problems. The bill applies to every industry imaginable and is not sufficiently limited to high-risk uses of ADT within those industries. Both the scope and various key terms (in particular consequential decisions, automated decision tools, algorithmic discrimination, and controlling factor), all require additional clarity and/or narrowing.
- Unworkable obligations in relation to the mandated impact assessments and notice requirements. In particular, the “opt-out” related requirements, including the requirement that businesses allow consumers to opt out of an ADT when “technically feasible”.
- Overly narrow confidentiality protections. Impact assessments must be exempt from the California Public Records Act to encourage candor and avoid concerns around assessments becoming fodder for litigation.
- Allowing multiple enforcement entities will invariably create confusion for compliance. Providing for a single enforcer (the Attorney General) will promote consistent interpretation and application across the state.
- Insufficient preemption protections and related concerns around regulatory activity. Preemption is needed both to prevent conflict with localities and, similarly, to prevent concerns around departments/agencies over-regulating this technology and getting ahead of the Legislature and Governor. These issues are too important to Californians across the state and our struggling economy to significantly delegate and defer to unelected officials.
Finally, a coalition of organizations, including Tech Equity, Oakland Privacy, and SEIU California, has adopted a support-if-amended position, seeking the following changes:
1. Revising the bill’s definitions (and its “controlling factor” requirement, in particular) to prevent deployers from avoiding compliance with the bill simply by having a human rubber stamp algorithmic recommendations;
2. Excluding worker management and other issues related to ongoing employment because of the diverse sets of laws that apply, the unique power dynamics, and the wide range of harms that go beyond discrimination that can occur when AI is deployed in the workplace;
3. Strengthening the notice and explanation provisions so that workers and consumers have a meaningful understanding of how and why they are assessed by AI systems;
4. Strengthening the impact assessment provisions by requiring unbiased, independent auditors and that all forms of discrimination must be assessed and mitigated;
5. Ensuring the bill has adequate enforcement remedies and does not undermine existing laws.
As described below, author’s amendments respond to 1), 3), and 5).
6) Author’s amendments. In response to feedback from various stakeholders, the author has offered several amendments to clarify and strengthen the provisions of this bill. In broad strokes, the amendments would do the following:
- Clarify the categories of personal information in impact assessments in order to align with California Consumer Privacy Act:
(3) A summary of the type of data categories of information collected from natural persons and processed by the automated decision tool when it is used to make, or be a controlling substantial factor in making, a consequentialdecision. decision, including, but not limited to, all of the following:
(A) Each category of personal information identified by reference to the applicable subparagraph enumerated under paragraph (1) of subdivision (v) of Section 1798.140 of the Civil Code.
(B) Each category of sensitive personal information identified by reference to the applicable paragraph and subparagraph enumerated under subdivision (ae) of Section 1798.140 of the Civil Code.
(C) Each category of information related to a natural person’s receipt of sensitive services, as defined in Section 56.05 of the Civil Code, identified by reference to the specific category of sensitive service enumerated in the definition.
- Require a deployer to provide natural persons with sufficient information to better enable them to take advantage of the alternative selection process or accommodation, in lieu of being subject to a decision made by an ADT:
(D) Information sufficient to enable the natural person to request to be subject to an alternative selection process or accommodation, as applicable, in lieu of the automated decision tool, as provided in subdivision (b).
- Require developers to provide deployers with a description of the deployers’ responsibilities under the bill, whenever an ADT is transferred from a developer to a deployer. This will foster compliance with the bill’s requirements:
(4) A description of the deployer’s responsibilities under this chapter.
- Require impact assessments to be maintained for five years, rather than two years. This will create a more robust evidentiary record for any possible enforcement actions under the bill:
(6) Maintain for two five years after completion the results of an impact assessment.
- Permit the Civil Rights Department to provide an impact assessment it receives to a public prosecutor for the purposes of assisting that prosecutor in initiating or litigating a civil action:
(c) The Civil Rights Department may provide an impact assessment it receives to a public prosecutor listed in subdivision (a) of Section 22756.8 to assist that public prosecutor in initiating or litigating a civil action under that section
- Grant the Civil Rights Department the authority to bring a civil action against a deployer or developer. This change fosters a more complete enforcement scheme:
22756.8. (a) (1) Any of the following public attorneys entities may bring a civil action against a deployer or developer for a violation of this chapter:
(A) The Attorney General in the name of the people of the State of California.
(B) A district attorney, county counsel, or city attorney for the jurisdiction in which the violation occurred.
(C) A city prosecutor in any city having a full-time city prosecutor, with the consent of the district attorney.
(D) The Civil Rights Department.
- Clarify that the protections offered under the bill are not intended to undermine or overwrite existing rights and remedies. Although nothing in the bill in print suggests otherwise, this has been a point of confusion for many stakeholders:
22756.11. The rights, remedies, and penalties established by this chapter are cumulative and shall not be construed to supersede the rights, remedies, or penalties established under other laws including, but not limited to, Chapter 6 (commencing with Section 12940) of Part 2.8 of Division 3 of Title 2 of the Government Code and Section 51 of the Civil Code.
- Replace the phrase “controlling factor” with “substantial factor” throughout the bill for purposes of determining when an artificial intelligence system rises to the level of an ADT subject to the bill. A number of stakeholders have expressed concerns that “controlling” sets too high of a bar, excluding circumstances in which an ADT may have influenced but not decided the outcome. On the other hand, the author does not want to capture circumstances in which an algorithm plays a minor role that is incapable of altering the outcome of the decision:
(c) “Automated decision tool” means a system or service that uses artificial intelligence and has been specifically developed and marketed to, or specifically modified to, make, or be a controlling substantial factor in making, consequential decisions.
- Clarify that developers are prohibited from making available discriminatory ADTs. Previously, while developers were required to run impact assessments, there was no clear prohibition on the release of discriminatory ADTs:
(b) A developer shall not make available to potential deployers an automated decision tool that results in algorithmic discrimination.
- Make various other small changes to improve clarity.
7) Additional considerations. Going forward, there are two key issues that the author may wish to address.
Application of administrative penalties. While this bill empowers the Civil Rights Department to bring administrative enforcement actions against developers or deployers that violate certain provisions, committee staff have been informed that the Civil Rights Department does not currently possess the necessary staff or resources to conduct administrative enforcement hearings. At a minimum, this would likely require new positions for administrative law judges to preside over the hearings as well as an initial investment to adopt and implement procedures for the presentation of evidence, arguments, and testimony related to an alleged violation. Going forward, the author may wish to examine whether the bill should expressly address this issue.
Relationship to the CCPA and CPPA. Proposition 24 (2020) created the California Privacy Protection Agency (Privacy Agency) in California, vested with full administrative power, authority, and jurisdiction to implement and enforce the CCPA and the CPRA. The Agency’s responsibilities include updating existing regulations, and adopting new regulations.
In March 2020, the Privacy Agency issued draft regulations governing the use of automated decision-making technology (ADMT) by businesses that are subject to the CCPA.[13] These regulations would require businesses that use ADMTs for “significant decisions” to: conduct and submit risk assessments to the Privacy Agency; provide consumers pre-use notice, the right to opt out, and certain information about the ADMT; and prohibit discrimination based on protected classes for businesses that use biometric data for significant decisions. Thus, it appears that there are significant overlaps between the proposed regulations and some provisions of this bill with respect to risk assessments, pre-use notice, and the right to opt out of an automated decision-making process. Going forward, the author may wish to examine ways of aligning these two parallel efforts at regulating automated decisionmaking processes.
8) Related legislation. AB 2885 (Bauer-Kahan, 2024) would establish a uniform definition for “artificial intelligence” in California’s codes.
AB 331 (Bauer-Kahan, 2023) was substantially similar to this bill. The bill was held in Assembly Appropriations.
AB 302 (Ward, Chap. 800, Stats. 2023) required the California Department of Technology (CDT) to conduct an inventory of all high-risk automated decision systems being used in state agencies. The bill was chaptered on October 8th, 2023.
SB 313 (Dodd, 2023) would have established an Office of Artificial Intelligence within CDT, with “the powers and authorities necessary to guide the design, use, or deployment of automated systems by a state agency to ensure that all AI systems are designed and deployed in a manner that is consistent with state and federal laws and regulations regarding privacy and civil liberties and that minimizes bias and promotes equitable outcomes for all Californians.” The bill was held in Senate Appropriations.
AB 2269 (Chau, 2020) would have established the Automated Decision Systems (ADS) Accountability Act of 2020. The Act would have required businesses in California that provide a person, as defined, with a program or device that uses an automated decision system (ADS) to take affirmative steps to ensure that there are processes in place to continually test for biases during the development and usage of the ADS, conduct an ADS impact assessment on its program or device to determine whether the ADS has a disproportionate adverse impact on a protected class, as specified, examine if the ADS in question serves reasonable objectives and furthers a legitimate interest, and compare the ADS to alternatives or reasonable modifications that may be taken to limit adverse consequences on protected classes. The bill was held in Assembly Privacy and Consumer Protection.
ARGUMENTS IN SUPPORT:
Microsoft Corporation states:
AB 2930 aligns with the Microsoft Responsible AI framework and recognizes the promise and potential pitfalls that the next generation of AI may bring through the use of Automated Decision Tools (ADTs). The bill requires developers and users of ADTs to conduct and record impact assessments to evaluate the intended uses of ADTs including an analysis of potential adverse impact on the basis of race, color, ethnicity, sex, religion, age, national origin, or any other classification protected by state law. We believe AB 2930 establishes appropriate legal guardrails on the development and deployment of ADT systems through the use of impact assessments and design evaluations.
Secure Justice states:
We believe AB 2930 is a critical need in this fast-emerging world of AI decision making. Sufficient research exists to demonstrate that many individuals are being actively harmed today as AI proceeds without sufficient oversight, transparency into the decision making, and lack of accountability when discriminatory decisions occur. AB 2930 addresses each of these concerns.
Workday states:
We believe that AB 2930 would establish a strong, pragmatic, and thoughtful regulatory framework for high-risk AI in California. Given the pace of innovation and AI adoption, we believe a “wait and see” approach when it comes to AI and public policy is unworkable and sensible guardrails are needed now to address the AI trust gap. Similarly, we believe that attempting to address every policy issue raised by AI’s development and deployment in a single legislative measure too often leads to proposals that collapse under their own weight and result in inaction. AB 2930 is a sensible way forward that addresses algorithmic discrimination, establishes rigorous AI governance requirements, and builds public trust in AI.
ARGUMENTS IN OPPOSITION:
California Chamber of Commerce writes:
We agree that it is critical that companies take care to reduce bias and discrimination in consequential decisions impacting people. At the same time, we believe these issues exist whether these decisions are human made from start to finish, or a biproduct of using or incorporating new technologies in the decision-making process. As such, it is critical that any regulatory efforts proceed with caution and precision, particularly as technology is still developing and has the potential to reduce, if not one day eliminate, such undesirable outcomes.
Electronic Frontier Foundation writes:
A.B. 2930, in covering private and government entities and several areas of consequential decisions, seeks to offer a one-size-fits all solution to an issue that is too complex for a single set of standards. While there may be general common principles for evaluating automated decisionmaking tools, implementing such standards depends heavily on context.
Technet writes:
While existing supervisory risk management and operational resiliency expectations may not expressly address AI, existing risk management principles provide a framework for financial institutions implementing AI to operate in a safe, sound and fair manner – as they would for the use of any other technology. Because AB 2930 is duplicative of the robust rules that our licensed members already follow, and due to the potential negative impacts to California consumers, we must respectfully oppose AB 2930.
REGISTERED SUPPORT / OPPOSITION:
Support
Microsoft Corporation
Secure Justice
Workday
Support If Amended
American Federation of Musicians, Local 7
Bsa the Software Alliance
California Employment Lawyers Association
California Nurses Association
Center for Democracy and Technology
Center on Race and Digital Justice
Consumer Reports
East Bay Community Law Center
Economic Security California Action
Equal Rights Advocates
Greenlining Institute
Legal Aid At Work
Oakland Privacy
Rise Economy
Seiu California
Techeqiuty Action
Opposition
American Council of Life Insurers
American Property Casualty Insurance Association
American Staffing Association
Association of California Life and Health Insurance Companies
Association of National Advertisers
California Association of Realtors
California Bankers Association
California Chamber of Commerce
California Community Banking Network
California Credit Union League
California Financial Services Association
California Hotel & Lodging Association
California Manufacturers & Technology Association
California Mortgage Bankers Association
California Retailers Association
California Staffing and Recruiting Association
Chamber of Progress
Civil Justice Association of California
Computer and Communications Industry Association
CTIA
Electronic Frontier Foundation
Insights Association
Los Angeles Area Chamber of Commerce
Mortgage Bankers Association
National Association of Mutual Insurance Companies
Personal Insurance Federation of California
State Privacy and Security Coalition, INC.
Technet
Oppose Unless Amended
Internet.works
Analysis Prepared by: Slater Sharp / P. & C.P. / (916) 319-2200
Santariano & Metz, “Using A.I. to Detect Breast Cancer That Doctors Miss,” New York Times, Mar. 5, 2023, https://www.nytimes.com/2023/03/05/technology/artificial-intelligence-breast-cancer-detection.html. ↑
Broussard, “Artificial Unintelligence,” MIT Press, 2019. ↑
McKinsey & Co., “The state of AI in 2022—and a half decade in review,” Dec. 6, 2022, https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2022-and-a-half-decade-in-review. ↑
Rohit Sehgal, “AI Needs Data More Than Data Needs AI”, Forbes, Oct. 5, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/10/05/ai-needs-data-more-than-data-needs-ai/. ↑
Aditya Malik, “AI Bias In Recruitment: Ethical Implications And Transparency,” Forbes, Sep. 25, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/09/25/ai-bias-in-recruitment-ethical-implications-and-transparency/. ↑
Jeffrey Dastin, “Amazon scraps secret AI recruiting tool that showed bias against women,” Reuters, Oct. 9, 2018, https://www.reuters.com/article/amazoncom-jobs-automation/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSL2N1VB1FQ/. ↑
Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, “Machine Bias”, Propublica, May 23, 2016, https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” California. ↑
Caleb J Colón-Rodríguez, “Shedding Light on Healthcare Algorithmic and Artificial Intelligence Bias,” U.S. Department of Health & Human Services Office of Minority Health, Jul. 12, 2023, minorityhealth.hhs.gov/news/shedding-light-healthcare-algorithmic-and-artificial-intelligence-bias. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” California. ↑
Emmanuel Martinez and Lauren Kirchner, “The Secret Bias Hidden in Mortgage-Approval Algorithms,” Markup, Aug. 25, 2021, themarkup.org/denied/2021/08/25/the-secret-bias-hidden-in-mortgage-approval-algorithms. ↑
The White House, “Blueprint for an AI Bill of Rights,” Oct. 2022, https://www.whitehouse.gov/ostp/ai-bill-of-rights/. ↑
California Privacy Protection Agency, “Draft Risk Assessment and Automated Decisionmaking Technology Regulations,” Mar. 2024, https://cppa.ca.gov/meetings/materials/20240308_item4_draft_risk.pdf. ↑
Discussed in Hearing
Senate Standing Committee on Appropriations
Senate Standing Committee on Judiciary
Assembly Floor
Assembly Standing Committee on Judiciary
Assembly Standing Committee on Privacy and Consumer Protection
Bill Author
Bill Co-Author(s):