(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill would, until July 1, 2030, instead require any state agency authorized to provide or enter into contracts relating to public benefit programs, or any local government agency authorized to provide or enter into contracts relating to public benefit programs funded by state funds, as specified, to provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California. The bill would also prohibit a state agency or specified local agency from using, or contracting with a call center that uses, artificial intelligence (AI) or automated decision systems (ADS) that would eliminate or automate core job functions of a worker, as specified. The bill would require an agency that utilizes AI or ADS that impact core job functions of workers to notify the workers, their collective bargaining representatives, and the public within a specified timeframe about prescribed information, including a general description of the AI or ADS system. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would, until July 1, 2030, delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would, until July 1, 2030, make the above-described provisions inapplicable to a designated public hospital system, a health care district, or a health care service plan that contracts with the State Department of Health Care Services, as specified.

This bill would, until July 1, 2030, extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to specified contracts or subcontracts, including, among others, extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025, and contracts by a county human services agency for translation or interpretation services.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill would, until July 1, 2030, instead require any state agency authorized to provide or enter into contracts relating to public benefit programs, or any local government agency authorized to provide or enter into contracts relating to public benefit programs funded by state funds, as specified, to provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California. The bill would also prohibit a state agency or specified local agency from using, or contracting with a call center that uses, artificial intelligence (AI) or automated decision systems (ADS) that would eliminate or automate core job functions of a worker, as specified. The bill would require an agency that utilizes AI or ADS that impact core job functions of workers to notify the workers, their collective bargaining representatives, and the public within a specified timeframe about prescribed information, including a general description of the AI or ADS system. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would, until July 1, 2030, delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would, until July 1, 2030, extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to specified contracts or subcontracts, including, among others, extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025, and contracts by a county human services agency for translation or interpretation services.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill would would, until July 1, 2030, instead require any state agency authorized to provide or enter into contracts relating to public benefit programs, or any local government agency authorized to provide or enter into contracts relating to public benefit programs funded by state funds, as specified, to provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California. The bill would also prohibit a state agency or specified local agency from using, or contracting with a call center that uses, artificial intelligence (AI) or automated decision systems (ADS) that would eliminate or automate core job functions of a worker, as specified. The bill would require an agency that utilizes AI or ADS that impact core job functions of workers to satisfy specified requirements, including developing an impact assessment report, as prescribed. notify the workers, their collective bargaining representatives, and the public within a specified timeframe about prescribed information, including a general description of the AI or ADS system. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would would, until July 1, 2030, delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would would, until July 1, 2030, extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to specified contracts or subcontracts, including including, among others, extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025. 2025, and contracts by a county human services agency for translation or interpretation services.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill would instead require any state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services, or any local government agency authorized to provide or enter into contracts relating to public benefit programs funded by state funds to provide services related to public benefits or services, funds, as specified, to provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California. The bill would also prohibit a state agency or specified local agency from using, or contracting with a call center that uses, artificial intelligence (AI) or automated decision systems (ADS) that would eliminate or automate core job functions of a worker, as specified. The bill would require an agency that utilizes AI or ADS that impact core job functions of workers to satisfy specified requirements, including developing an impact assessment report, as prescribed. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill would require any state agency is authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services, this bill would require that agency to provide or contract for or any local government agency funded by state funds to provide services related to public benefits or services, as specified, to provide services through, or contract for services provided by by, a call center that directly serves callers with services performed solely with and by workers employed in California, as specified. California. The bill would also prohibit a state agency or specified local agency from using using, or contracting with a call center that uses, artificial intelligence (AI), automated decision systems (ADS), or other similar technology (AI) or automated decision systems (ADS) that would eliminate or automate essential core job functions of a worker. worker, as specified. The bill would require an agency that utilizes AI, ADS, or other similar technologies AI or ADS that impact essential core job functions of workers to satisfy specified requirements, including developing an impact assessment report, as prescribed. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

If a state agency is authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services, this bill would require that agency to provide or contract for services provided by a call center that directly serves callers with services performed solely with and by workers, as defined, workers employed in California, as specified. The bill would also prohibit a state agency or specified local agency from using artificial intelligence (AI), automated decision systems (ADS), or other similar technology that would eliminate or automate essential job functions of a worker. The bill would require an agency that utilizes AI, ADS, or other similar technologies that impact essential job functions of workers to satisfy specified requirements, including developing an impact assessment report, as prescribed. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

(1)Existing law prohibits, with specified exceptions, a state agency authorized to enter into contracts relating to public benefit programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed under that contract, will be performed solely with workers employed in California. Existing law provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance. Existing law also authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

If a state agency is authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services, this bill would require that agency to provide or contract for services provided by a call center that directly serves callers with services performed solely with and by workers, as defined, employed in California, as specified. The bill would require a contractor to certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. The bill would also extend these contracting requirements to local government agencies. By imposing new requirements on local government agencies, this bill would impose a state-mandated local program.

This bill would delete the above-described exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.

This bill would extend the right to terminate a contract to local government agencies. The bill would also extend the right to terminate a contract in the case that the contractor or subcontractor does not employ workers.

This bill would provide that the bills provisions would not apply to contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.

(2)The bill would include findings that changes proposed by this bill address a matter of statewide concern rather than a municipal affair and, therefore, apply to all cities, including charter cities.

(3)The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.

This bill would provide that, if the Commission on State Mandates determines that the bill contains costs mandated by the state, reimbursement for those costs shall be made pursuant to the statutory provisions noted above.

 VETO  

Bill No: SB 1220

Author: Limón (D), et al.

Enrolled: 9/4/24

Vote: 27

SENATE GOVERNMENTAL ORG. COMMITTEE: 10-3, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Padilla, Portantino, Rubio, Smallwood-Cuevas

NOES: Jones, Ochoa Bogh, Seyarto

NO VOTE RECORDED: Ashby, Nguyen, Roth

SENATE LOCAL GOVERNMENT COMMITTEE: 5-1, 4/17/24

AYES: Durazo, Glazer, Skinner, Wahab, Wiener

NOES: Seyarto

NO VOTE RECORDED: Dahle

SENATE APPROPRIATIONS COMMITTEE: 5-2, 5/16/24

AYES: Caballero, Ashby, Becker, Bradford, Wahab

NOES: Jones, Seyarto

SENATE FLOOR: 32-7, 5/21/24

AYES: Alvarado-Gil, Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NOES: Dahle, Grove, Jones, Nguyen, Niello, Ochoa Bogh, Seyarto

NO VOTE RECORDED: Allen

SENATE FLOOR: 31-8, 8/30/24

AYES: Allen, Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NOES: Alvarado-Gil, Dahle, Grove, Jones, Nguyen, Niello, Ochoa Bogh, Seyarto

NO VOTE RECORDED: Glazer

ASSEMBLY FLOOR: 61-8, 8/29/24 - See last page for vote

SUBJECT: Public benefits contracts: phone operator jobs

SOURCE: California Labor Federation

Service Employees International Union

DIGEST:

This bill prohibits state and local agencies from contracting for using call center services related to public benefits that use artificial intelligence (AI) or automated decision-making systems (ADS) that eliminate or automate workers’ core functions, with certain exceptions.

ANALYSIS:

Existing law:

1. Prohibits, with specified exemptions, a state agency authorized to enter into contracts relating to public benefit programs, as defined, from contracting for services provided by a call center that directly serves applicants for, recipients for, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed will be performed solely with workers employed in California.
2. Defines “public benefit programs,” for the above provisions to mean California Work Opportunity and Responsibility to Kids (CalWORKS), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.
3. Defines “call center” for the above provisions to mean a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.
4. Provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
5. Authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill:

1. Prohibits state and local agencies from contracting for or using call center services related to public benefits that use AI or ADS that eliminate or automate the core job functions of a worker.
2. Requires such agencies, before using or contracting for the use of AI or ADS that impact the core job function, as defined, of a worker, to first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a description of the AI or ADS and the intended purpose for the AI or ADS.
3. Exempts form the notice requirement, the following uses of AI or ADS:
   1. Phone trees or prompts that route callers to appropriate staff, information, or options.
   2. Tools that support workers by providing suggestions, AI-powered searches and support of policy, procedural, or training information.
   3. AI used by workers to assist callers with public benefits or assist workers in service delivery.
4. Extends to local governments the existing requirement that, for state agency contracts for call center services related to public benefits programs, the contractor certify in their bid that the service will be performed solely by workers employed in California.
5. Defines “core job functions” to mean a component of call service delivery that requires the exercise of judgment and discretion to successfully accomplish a complex task.
6. Defines “public benefits program to mean the California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, Covered California, Employment Development Department Insurance, Child Welfare Services, California Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), and other public benefits provided by a local government agency funded in any part by state funding from the California Health and Human Services Agency to provide the same or similar benefits.
7. Exempts, among other things, the following:
   1. Emergency services such as 988 and 911.
   2. Call center services provided for a local public benefit program in response to an emergency or disaster, including mutual aid, or temporary staff relocations during emergency or disaster, and until normal operations have been restored.
   3. A contract or subcontract by a county human services agency for translation or interpretation services.
   4. Specified call centers in public hospitals and healthcare district.
8. Provides that the provisions of this bill do not apply to contracts related to a request for proposal or bid that occurred before January 1, 2025.
9. Includes a sunset date of July 1, 2030.

Background

Author’s Statement. According to the author’s office, “SB 1220 will prohibit the state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or other technology that completely replaces workers. This bill defines a worker as a person. It does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI. The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI.”

Artificial Intelligence. AI refers to the development of computer systems that can perform tasks typically requiring human intelligence. These tasks include learning, reasoning, problem-solving, perception, and natural language understanding. AI enables computers to learn from data and experiences, allowing them to handle complex tasks without explicit programming. AI has the potential to revolutionize numerous sectors, including healthcare, finance, and transportation, by improving efficiency, reducing costs, and enhancing decision-making capabilities.

One of the most advanced and well-known AI systems, GPT-4, was developed by OpenAI and released on Pi Day (3/14), 2023. GPT-4 has demonstrated remarkable capabilities compared to its predecessor, GPT-3.5, which had been released to the world just six months prior. For example, GPT-4 scored around the top 10% on a simulated bar exam, while GPT-3.5 scored around the bottom 10%.

GPT-4 also scored a five out of five on several Advanced Placement (AP) exams, including Art History, Biology, Environmental Science, Macro- and Microeconomics, and Psychology, showcasing a knowledge across a diverse range of subjects. GPT-4 is particularly impressive at reasoning, scoring in the 93^rd percentile on the SAT Evidence-Based Reading & Writing exam and in the 99^th percentile on the Graduate Record Examination (GRE) Verbal exam.

The remarkable advancements in AI, exemplified by GPT-4's impressive performance on various professional and academic benchmarks, showcase its potential for practical applications in diverse fields. From law and education to science and technology, AI's ability to understand complex tasks and generate nuanced responses could lead to transformative solutions and new insights across multiple disciplines.

There are different types of AI, such as rule-based systems to make decisions or carry out tasks, and machine learning systems, which learn from data and experience to improve their performance over time. Deep learning, a type of machine learning, utilizes neural networks that mimic the structure and function of the human brain. Neural networks are used in deep learning to process large amounts of data and learn to recognize patterns, make predictions, and perform other tasks with a high degree of accuracy.

AI is already being used in a wide range of applications, including virtual assistants, image and speech recognition, fraud detection, predictive analytics, and autonomous vehicles. AI excels at fraud detection and predictive analytics due to its ability to rapidly and accurately analyze large datasets, detect patterns and anomalies challenging for humans to discern, and learn from historical data to predict future events. By detecting potential fraudulent activity and predicting future outcomes, AI can help businesses and governments make more informed decisions, save time and resources, and prevent financial losses.

While AI has the potential to revolutionize government services and decision-making, it also presents several challenges and concerns. One major issue is the potential for bias in AI systems. AI algorithms are only as unbiased as the data they are trained on, and if that data is biased or incomplete, it can result in discriminatory outcomes. For example, if an AI system is used to assess job applicants, it may inadvertently discriminate against certain groups if the data used to train the system is biased towards certain demographics. This could perpetuate existing inequalities in the workplace. To minimize bias in AI systems, it is crucial to ensure that the data used to train these systems is representative of the target population.

Another concern is the potential for privacy violations. AI systems can collect and process vast amounts of personal data, and if not properly secured, that data could be vulnerable to misuse or hacking. This is particularly concerning in government contexts, where sensitive information such as criminal records, medical histories, and financial information is often collected and stored. Additionally, there is concern that AI could displace jobs in certain sectors, particularly those that rely heavily on repetitive or routine tasks. While AI has the potential to automate many tasks and increase efficiency, it could also lead to significant job losses and create new inequalities in the workforce.

Further, there is the risk of unintended consequences. AI systems are complex and difficult to fully understand, and it is possible that they could produce unexpected or unintended outcomes. This is particularly concerning in government contexts where decisions made by AI systems could have far-reaching consequences for individuals and communities. For example, if an AI system is used to determine eligibility for social services, errors or biases in the system could result in individuals being denied important benefits. As such, it is important that AI is developed and implemented in a responsible and ethical manner, with proper safeguards and oversight in place to mitigate these risks.

AI in Government Services. AI is already being used in various ways within California government, including in the areas of transportation, healthcare, public safety, natural resource management, and emergency preparation and response. For instance, in 2020, the University of California, San Diego (UCSD) developed an AI-powered tool that can analyze chest X-rays to detect potential cases of COVID-19. This tool helped healthcare professionals quickly identify patients who needed immediate care, leading to more efficient allocation of medical resources during the pandemic.

In 2019, the number of traffic fatalities in the United States was 36,096, which is more than 11.17 deaths per 100,000 population. These numbers underscore the critical need to improve traffic, pedestrian, and bicyclist safety. The AI Pedestrian Traffic Safety System – tested on 12 hours of traffic videos captured from existing traffic cameras in the city of Los Angeles – demonstrated more than 98% accuracy in counting pedestrians in real time. This project occurred as part of LA’s public commitment to the global initiative Vision Zero, which endeavors to reduce severe injuries and deaths in roadway collisions. Los Angeles’s specific Vision Zero goal is to eliminate all traffic deaths and serious injuries by 2025. At full scale, the system can cover all 56,000 miles of LA streets and in real time.

After the 2018 Camp Fire in California, AI technology was used to analyze satellite imagery and assess the extent of the damage caused by the fire. This information helped emergency responders prioritize their efforts, allocate resources more efficiently, and speed up the recovery process for affected communities.

However, it is important to acknowledge that there are potential risks and negative consequences associated with AI. Bias, privacy violations, job displacement, and unintended consequences are some of the concerns associated with AI. These issues must be addressed and mitigated to ensure that the benefits of AI are maximized while minimizing any potential risks.

In 2016, a study by ProPublica revealed that a widely used AI-based risk assessment tool, COMPAS, was biased against African-American defendants in predicting their likelihood of reoffending. The algorithm was found to be more likely to falsely flag black defendants as high-risk and white defendants as low-risk, perpetuating existing racial biases in the criminal justice system.

When Use of AI Goes Wrong. The author’s office points to a number of examples where AI or automation have replaced human workers and the results have been devastating. For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named Tessa. The version of Tessa that was initially tested and studied was a rule-based chatbot, meaning it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “we were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI, meaning it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began to telling callers to avoid “unhealthy foods” and only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400% interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93% of the AI’s fraud charges were wrong, however much of the damage had already been done.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes

According to the Assembly Appropriations Committee, costs to the state could reach tens of millions annually to expand current restrictions on out-of-state call centers to all public-benefit programs. Additionally, if the Commission on State Mandates determines the bill's requirements amount to a reimbursable state mandate, the state would need to offset costs to local agencies to procure new call center contracts

SUPPORT: (Verified 9/24/24)

California Labor Federation (co-source)

Service Employees International Union (co-source)

California Democratic Party

California School Employees Association, AFL-CIO

Orange County Employee Association

TechEquity Collaborative

211 San Diego

OPPOSITION: (Verified 9/24/24)

None received

ARGUMENTS IN SUPPORT: According to the California Labor Federation, “the growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. With the high volume of callers in need of support, public benefit programs such a Medi-Cal, the 988 Suicide Prevention & Mental Health Crisis Lifeline, and unemployment insurance assistance requires a support staff in a California that understands the issues facing the state’s most vulnerable populations. While not prohibiting the use of AI, SB 1220 will require agencies to produce impact assessment reports when utilizing AI, or ADS to guarantee that workers are at the center of the decision-making process during AI deployment.”

According to TechEquity Collaborative, “ultimately, AI simply cannot provide the level of empathy and effectiveness provided by a person. Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of needs, ranging from global pandemic to intense natural disasters, public benefit call center workers have stood with people in need of guidance. Without protections from AI, call center workers face the erosion and eventual elimination of their jobs.”

GOVERNOR'S VETO MESSAGE:

This bill prohibits state and local agencies from using public benefit-related call center services that use Al or ADS that eliminate or automate the core job function of a worker. This bill also extends to local governments an existing state requirement that public benefit-related call center services be performed solely by workers employed in California.

Technology can and should enhance the experience of the workforce - by making work more efficient and pushing us to attain new heights of achievement and innovation. At the same time, we must consider appropriate guardrails and control the risks posed by this technology.

On September 6, 2023, I signed Executive Order (EO) N-12-23 to underscore our commitment to developing a responsible process for the evaluation and deployment of Al within state government. Through the implementation of this EO, the state will soon issue criteria to evaluate the impact of Al on the state workforce, as well as guidelines on how state agencies and departments can support their employees.

Further, thanks to legislation enacted last year, my Administration is developing a comprehensive inventory of high-risk ADS that assist or replace human decision-making and significantly impact individuals. Analyzing these systems will help guide future actions and policies regarding the use of Al across the state, including in call centers for public benefit programs. Given that my Administration is actively undergoing efforts to identify, inventory, and analyze these systems, in addition to the efforts underway in my EO, imposing a prohibition on Al or ADS at this stage would be premature. For these reasons, I cannot sign this bill.

ASSEMBLY FLOOR: 61-8, 8/29/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Connolly, Mike Fong, Friedman, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Holden, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Luz Rivas, Rodriguez, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Valencia, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Robert Rivas

NOES: Megan Dahle, Davies, Essayli, Gallagher, Hoover, Mathis, Jim Patterson, Joe Patterson

NO VOTE RECORDED: Cervantes, Chen, Dixon, Flora, Lackey, Ortega, Sanchez, Ta, Villapudua, Zbur

Prepared by: Felipe Lopez / G.O. / (916) 651-1530

10/16/24 14:16:31

**** END ****

 UNFINISHED BUSINESS  

Bill No: SB 1220

Author: Limón (D), et al.

Amended: 8/19/24

Vote: 21

SENATE GOVERNMENTAL ORG. COMMITTEE: 10-3, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Padilla, Portantino, Rubio, Smallwood-Cuevas

NOES: Jones, Ochoa Bogh, Seyarto

NO VOTE RECORDED: Ashby, Nguyen, Roth

SENATE LOCAL GOVERNMENT COMMITTEE: 5-1, 4/17/24

AYES: Durazo, Glazer, Skinner, Wahab, Wiener

NOES: Seyarto

NO VOTE RECORDED: Dahle

SENATE APPROPRIATIONS COMMITTEE: 5-2, 5/16/24

AYES: Caballero, Ashby, Becker, Bradford, Wahab

NOES: Jones, Seyarto

SENATE FLOOR: 32-7, 5/21/24

AYES: Alvarado-Gil, Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NOES: Dahle, Grove, Jones, Nguyen, Niello, Ochoa Bogh, Seyarto

NO VOTE RECORDED: Allen

ASSEMBLY FLOOR: 61-8, 8/29/24 - See last page for vote

SUBJECT: Public benefits contracts: phone operator jobs

SOURCE: California Labor Federation

Service Employees International Union

DIGEST:

This bill prohibits state and local agencies from contracting for using call center services related to public benefits that use artificial intelligence (AI) or automated decision-making systems (ADS) that eliminate or automate workers’ core functions, with certain exceptions.

Assembly Amendments

(1) add additional exemptions to the bill; (2) delete the requirement that a state agency or local government must develop an impact assessment report when that state agency or local government enters into specified contracts related to AI or ADS; (3) add a sunset date of July 1, 2030; and (4) make other technical and clarifying amendments.

ANALYSIS:

Existing law:

1. Prohibits, with specified exemptions, a state agency authorized to enter into contracts relating to public benefit programs, as defined, from contracting for services provided by a call center that directly serves applicants for, recipients for, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed will be performed solely with workers employed in California.
2. Defines “public benefit programs,” for the above provisions to mean California Work Opportunity and Responsibility to Kids (CalWORKS), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.
3. Defines “call center” for the above provisions to mean a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.
4. Provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
5. Authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill:

1. Prohibits state and local agencies from contracting for or using call center services related to public benefits that use AI or ADS that eliminate or automate the core job functions of a worker.
2. Requires such agencies, before using or contracting for the use of AI or ADS that impact the core job function, as defined, of a worker, to first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a description of the AI or ADS and the intended purpose for the AI or ADS.
3. Exempts form the notice requirement, the following uses of AI or ADS:
   1. Phone trees or prompts that route callers to appropriate staff, information, or options.
   2. Tools that support workers by providing suggestions, AI-powered searches and support of policy, procedural, or training information.
   3. AI used by workers to assist callers with public benefits or assist workers in service delivery.
4. Extends to local governments the existing requirement that, for state agency contracts for call center services related to public benefits programs, the contractor certify in their bid that the service will be performed solely by workers employed in California.
5. Defines “core job functions” to mean a component of call service delivery that requires the exercise of judgment and discretion to successfully accomplish a complex task.
6. Defines “public benefits program to mean the California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, Covered California, Employment Development Department Insurance, Child Welfare Services, California Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), and other public benefits provided by a local government agency funded in any part by state funding from the California Health and Human Services Agency to provide the same or similar benefits.
7. Exempts, among other things, the following:
   1. Emergency services such as 988 and 911.
   2. Call center services provided for a local public benefit program in response to an emergency or disaster, including mutual aid, or temporary staff relocations during emergency or disaster, and until normal operations have been restored.
   3. A contract or subcontract by a county human services agency for translation or interpretation services.
   4. Specified call centers in public hospitals and healthcare district.
8. Provides that the provisions of this bill do not apply to contracts related to a request for proposal or bid that occurred before January 1, 2025.
9. Includes a sunset date of July 1, 2030.

Background

Author’s Statement. According to the author’s office, “SB 1220 will prohibit the state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or other technology that completely replaces workers. This bill defines a worker as a person. It does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI. The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI.”

Artificial Intelligence. AI refers to the development of computer systems that can perform tasks typically requiring human intelligence. These tasks include learning, reasoning, problem-solving, perception, and natural language understanding. AI enables computers to learn from data and experiences, allowing them to handle complex tasks without explicit programming. AI has the potential to revolutionize numerous sectors, including healthcare, finance, and transportation, by improving efficiency, reducing costs, and enhancing decision-making capabilities.

One of the most advanced and well-known AI systems, GPT-4, was developed by OpenAI and released on Pi Day (3/14), 2023. GPT-4 has demonstrated remarkable capabilities compared to its predecessor, GPT-3.5, which had been released to the world just six months prior. For example, GPT-4 scored around the top 10% on a simulated bar exam, while GPT-3.5 scored around the bottom 10%.

GPT-4 also scored a five out of five on several Advanced Placement (AP) exams, including Art History, Biology, Environmental Science, Macro- and Microeconomics, and Psychology, showcasing a knowledge across a diverse range of subjects. GPT-4 is particularly impressive at reasoning, scoring in the 93^rd percentile on the SAT Evidence-Based Reading & Writing exam and in the 99^th percentile on the Graduate Record Examination (GRE) Verbal exam.

The remarkable advancements in AI, exemplified by GPT-4's impressive performance on various professional and academic benchmarks, showcase its potential for practical applications in diverse fields. From law and education to science and technology, AI's ability to understand complex tasks and generate nuanced responses could lead to transformative solutions and new insights across multiple disciplines.

There are different types of AI, such as rule-based systems to make decisions or carry out tasks, and machine learning systems, which learn from data and experience to improve their performance over time. Deep learning, a type of machine learning, utilizes neural networks that mimic the structure and function of the human brain. Neural networks are used in deep learning to process large amounts of data and learn to recognize patterns, make predictions, and perform other tasks with a high degree of accuracy.

AI is already being used in a wide range of applications, including virtual assistants, image and speech recognition, fraud detection, predictive analytics, and autonomous vehicles. AI excels at fraud detection and predictive analytics due to its ability to rapidly and accurately analyze large datasets, detect patterns and anomalies challenging for humans to discern, and learn from historical data to predict future events. By detecting potential fraudulent activity and predicting future outcomes, AI can help businesses and governments make more informed decisions, save time and resources, and prevent financial losses.

While AI has the potential to revolutionize government services and decision-making, it also presents several challenges and concerns. One major issue is the potential for bias in AI systems. AI algorithms are only as unbiased as the data they are trained on, and if that data is biased or incomplete, it can result in discriminatory outcomes. For example, if an AI system is used to assess job applicants, it may inadvertently discriminate against certain groups if the data used to train the system is biased towards certain demographics. This could perpetuate existing inequalities in the workplace. To minimize bias in AI systems, it is crucial to ensure that the data used to train these systems is representative of the target population.

Another concern is the potential for privacy violations. AI systems can collect and process vast amounts of personal data, and if not properly secured, that data could be vulnerable to misuse or hacking. This is particularly concerning in government contexts, where sensitive information such as criminal records, medical histories, and financial information is often collected and stored. Additionally, there is concern that AI could displace jobs in certain sectors, particularly those that rely heavily on repetitive or routine tasks. While AI has the potential to automate many tasks and increase efficiency, it could also lead to significant job losses and create new inequalities in the workforce.

Further, there is the risk of unintended consequences. AI systems are complex and difficult to fully understand, and it is possible that they could produce unexpected or unintended outcomes. This is particularly concerning in government contexts where decisions made by AI systems could have far-reaching consequences for individuals and communities. For example, if an AI system is used to determine eligibility for social services, errors or biases in the system could result in individuals being denied important benefits. As such, it is important that AI is developed and implemented in a responsible and ethical manner, with proper safeguards and oversight in place to mitigate these risks.

AI in Government Services. AI is already being used in various ways within California government, including in the areas of transportation, healthcare, public safety, natural resource management, and emergency preparation and response. For instance, in 2020, the University of California, San Diego (UCSD) developed an AI-powered tool that can analyze chest X-rays to detect potential cases of COVID-19. This tool helped healthcare professionals quickly identify patients who needed immediate care, leading to more efficient allocation of medical resources during the pandemic.

In 2019, the number of traffic fatalities in the United States was 36,096, which is more than 11.17 deaths per 100,000 population. These numbers underscore the critical need to improve traffic, pedestrian, and bicyclist safety. The AI Pedestrian Traffic Safety System – tested on 12 hours of traffic videos captured from existing traffic cameras in the city of Los Angeles – demonstrated more than 98% accuracy in counting pedestrians in real time. This project occurred as part of LA’s public commitment to the global initiative Vision Zero, which endeavors to reduce severe injuries and deaths in roadway collisions. Los Angeles’s specific Vision Zero goal is to eliminate all traffic deaths and serious injuries by 2025. At full scale, the system can cover all 56,000 miles of LA streets and in real time.

After the 2018 Camp Fire in California, AI technology was used to analyze satellite imagery and assess the extent of the damage caused by the fire. This information helped emergency responders prioritize their efforts, allocate resources more efficiently, and speed up the recovery process for affected communities.

However, it is important to acknowledge that there are potential risks and negative consequences associated with AI. Bias, privacy violations, job displacement, and unintended consequences are some of the concerns associated with AI. These issues must be addressed and mitigated to ensure that the benefits of AI are maximized while minimizing any potential risks.

In 2016, a study by ProPublica revealed that a widely used AI-based risk assessment tool, COMPAS, was biased against African-American defendants in predicting their likelihood of reoffending. The algorithm was found to be more likely to falsely flag black defendants as high-risk and white defendants as low-risk, perpetuating existing racial biases in the criminal justice system.

When Use of AI Goes Wrong. The author’s office points to a number of examples where AI or automation have replaced human workers and the results have been devastating. For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named Tessa. The version of Tessa that was initially tested and studied was a rule-based chatbot, meaning it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “we were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI, meaning it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began to telling callers to avoid “unhealthy foods” and only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400% interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93% of the AI’s fraud charges were wrong, however much of the damage had already been done.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes

According to the Assembly Appropriations Committee, costs to the state could reach tens of millions annually to expand current restrictions on out-of-state call centers to all public-benefit programs. Additionally, if the Commission on State Mandates determines the bill's requirements amount to a reimbursable state mandate, the state would need to offset costs to local agencies to procure new call center contracts.

SUPPORT: (Verified 8/26/24)

California Labor Federation (co-source)

Service Employees International Union (co-source)

California Democratic Party

California School Employees Association, AFL-CIO

Orange County Employee Association

TechEquity Collaborative

211 San Diego

OPPOSITION: (Verified 8/26/24)

Nonce received

ARGUMENTS IN SUPPORT: According to the California Labor Federation, “the growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. With the high volume of callers in need of support, public benefit programs such a Medi-Cal, the 988 Suicide Prevention & Mental Health Crisis Lifeline, and unemployment insurance assistance requires a support staff in a California that understands the issues facing the state’s most vulnerable populations. While not prohibiting the use of AI, SB 1220 will require agencies to produce impact assessment reports when utilizing AI, or ADS to guarantee that workers are at the center of the decision-making process during AI deployment.”

According to TechEquity Collaborative, “ultimately, AI simply cannot provide the level of empathy and effectiveness provided by a person. Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of needs, ranging from global pandemic to intense natural disasters, public benefit call center workers have stood with people in need of guidance. Without protections from AI, call center workers face the erosion and eventual elimination of their jobs.”

ASSEMBLY FLOOR: 61-8, 8/29/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Connolly, Mike Fong, Friedman, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Holden, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Luz Rivas, Rodriguez, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Valencia, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Robert Rivas

NOES: Megan Dahle, Davies, Essayli, Gallagher, Hoover, Mathis, Jim Patterson, Joe Patterson

NO VOTE RECORDED: Cervantes, Chen, Dixon, Flora, Lackey, Ortega, Sanchez, Ta, Villapudua, Zbur

Click here to enter text.

Prepared by: Felipe Lopez / G.O. / (916) 651-1530

8/29/24 20:03:39

**** END ****

SENATE THIRD READING

STR Bill Id:SB 1220

Author:(Limón)

As Amended Ver:August 19, 2024

Majority vote

SUMMARY

Extends to local governments the existing prohibition on outsourcing public contracts for call services related to public benefits. Prohibits state and local agencies from contracting for or using call center services related to public benefits that use artificial intelligence (AI) or automated decisionmaking systems (ADS) that eliminate or automate workers' core job functions, with certain exceptions. Requires, before an agency may use or contract for the use of AI or ADS that impacts the core job function of a worker, that the agency first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system. Sunsets July 1, 2030.

Major Provisions

1. Extends to local governments the existing requirement that, for state agency contracts for call center services related to public benefits programs, the contractor certify in their bid that the services will be performed solely by workers employed in California.
2. Prohibits state and local agencies from contracting for or using call center services related to public benefits that use AI or ADS that eliminate or automate the core job function of a worker. A noncompliant contractor may be penalized for the amount the contractor or subcontractor paid for the cost of the AI or ADS to perform core job functions that were eliminated or automated.
3. Requires such agencies, before using or contracting for the use of AI or ADS that impact the core job function of a worker, to first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system.
4. Exempts from the notice requirement, the following uses of AI or ADS:
   1. Phone trees or prompts that route callers to appropriate staff, information, or options.
   2. Tools that support workers by providing suggestions, AI-powered searches and support of policy, procedural, or training information.
   3. AI used by workers to assist callers with public benefits or assist workers in service delivery.
5. Defines, among other terms:
   1. "Core job function" as a component of call service delivery that requires the exercise of judgment and discretion to successfully accomplish a complex task.
   2. "Public benefits program" as California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, Covered California, Employment Development Department Insurance, Child Welfare Services, California Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), and other public benefits provided by a local government agency funded in any part by state funding from the California Health and Human Services Agency to provide the same or similar benefits.
6. Exempts, among other things:
   1. Emergency services such as 988 and 911.
   2. Call center services provided for a local public benefit program in response to an emergency or disaster, including mutual aid, or temporary staff relocations during the emergency or disaster, and until normal operations have been restored.
   3. A contract or subcontract by a county human services agency for translation or interpretation services.
   4. Specified call centers in public hospitals and healthcare districts.
7. Provides that it does not apply to contracts related to a request for proposal or bid that occurred before January 1, 2025.
8. Sunsets July 1, 2030.

COMMENTS

Background. With the arrival of the latest generation of large language models such as ChatGPT-4, chatbots have infiltrated virtually every white collar professional setting. Such models can process enormous datasets, identify hidden patterns, make decisions with a high degree of accuracy and reliability, and converse fluidly with humans. The widespread adoption of this technology, which may continue to develop at an explosive pace, carries the potential to disrupt jobs in virtually every sector of the economy. Goldman Sachs recently predicted that roughly two-thirds of U.S. occupations could see a quarter to half of their workload displaced by AI.

Especially vulnerable are call center jobs, which can offer decent salaries, commission, and benefits, and which often do not require a college degree—providing a path to upward mobility for individuals with disadvantaged backgrounds. While AI currently falls well short of the emotional and intellectual complexity that these services frequently demand, AI can adequately address many routine customer service problems at a fraction of the cost of employing a human. For many companies, the potential diminishment in quality of customer service is worth the savings.

This is not the first time call centers have faced the threat of outsourcing: when the Schwarzenegger and Davis Administrations approved out-of-country vendor contracts for CalFRESH call centers, California enacted protections against outsourcing for call center workers who provide services related to public benefits. As the executive branch has begun procuring generative AI tools for various functions, including call center services for tax guidance, this bill seeks to extend those protections to address the threat of automation by AI.

This bill would, among other things, extend to local governments the existing prohibition on outsourcing public contracts for call services related to public benefits. The bill would also prohibit state and local agencies from contracting for or using call center services related to public benefits that use AI or ADS that eliminate or automate workers' core job functions, with certain exceptions. Before an agency may use or contract for the use of an AI or ADS that impacts the core job function of a worker, the agency must first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system. The bill's provisions sunset July 1, 2030.

According to the Author

The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving the most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not artificial intelligence.

Arguments in Support

SEIU California, a co-sponsor of the bill, writes:

While AI and automated systems can be very helpful in helping streamlining workload and assisting public employees in serving higher caseloads than they could on their own it is the duty of the State to regulate the use and purpose of technology. Especially in highly sensitive settings, such as safety net services where the public are already in an extremely vulnerable position.

AI and automated systems run on often unknown algorithms fed with data that may be unreliable or replicate unwanted interactions. This creates a level of risk that can be particularly harmful to individuals in dire need of benefits and services like unemployment or mental health services.

The growth of AI has posed many challenges to California's workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. Without worker engagement implementation can be disastrous, with worker engagement even disastrous implementations can overtime serve the purpose needed. That distinction is one that this legislation underscores.

For instance, the inaugural role out of CalSAWs only had limited worker engagement, in its initial implementation it significantly increased workload, delayed benefits for recipients, putting counties and the most vulnerable populations into crisis. Workers have since been included in every transition and have helped turn the use of CalSAWs into a tool that is now being utilized by almost every county. Our caseloads to worker ratio in almost every single agency are well above recommended guidelines, the use and role of technology exists but that value added is only when the workers are at the helm.

Arguments in Opposition

Several 211 providers opposed a prior version of the bill unless amended to remove reference to 211—a change that was made in the Privacy and Consumer Protection Committee. Additionally, trade associations for health care services plans opposed the bill unless it was amended to restore an exemption in existing law for contracts under control of the Department of Managed Health Care—another change made in the Privacy and Consumer Protection Committee. Finally, the Association of California Healthcare Districts and California Association of Public Hospitals and Health Systems have submitted a letter of concern seeking exemptions. These concerns appear to have been addressed by amendments made by the Assembly Appropriations Committee.

FISCAL COMMENTS

According to the Assembly Appropriations Committee, costs to the state could reach tens of millions annually to expand current restrictions on out-of-state call centers to all public-benefit programs. Additionally, if the Commission on State Mandates determines the bill's requirements amount to a reimbursable state mandate, the state would need to offset costs to local agencies to procure new call center contracts.

VOTES

SENATE FLOOR: 32-7-1

YES: Alvarado-Gil, Archuleta, Ashby, Atkins, Becker, Blakespear, Bradford, Caballero, Cortese, Dodd, Durazo, Eggman, Glazer, Gonzalez, Hurtado, Laird, Limón, McGuire, Menjivar, Min, Newman, Padilla, Portantino, Roth, Rubio, Skinner, Smallwood-Cuevas, Stern, Umberg, Wahab, Wiener, Wilk

NO: Dahle, Grove, Jones, Nguyen, Niello, Ochoa Bogh, Seyarto

ABS, ABST OR NV: Allen

ASM PUBLIC EMPLOYMENT AND RETIREMENT: 5-0-2

YES: McKinnor, Boerner, Hart, Stephanie Nguyen, Luz Rivas

ABS, ABST OR NV: Lackey, Alanis

ASM PRIVACY AND CONSUMER PROTECTION: 7-2-2

YES: Bauer-Kahan, Bryan, Irwin, Lowenthal, Ortega, Ward, Wilson

NO: Dixon, Hoover

ABS, ABST OR NV: Joe Patterson, Wicks

ASM APPROPRIATIONS: 11-3-1

YES: Wicks, Arambula, Bryan, Calderon, Wendy Carrillo, Mike Fong, Grayson, Haney, Hart, Pellerin, Villapudua

NO: Sanchez, Dixon, Ta

ABS, ABST OR NV: Jim Patterson

UPDATED

VERSION: August 19, 2024

CONSULTANT: Josh Tosney / P. & C.P. / (916) 319-2200 FN: 0004149

Date of Hearing: August 7, 2024

ASSEMBLY COMMITTEE ON APPROPRIATIONS

Buffy Wicks, Chair

SBPCA Bill Id:SB 1220

Author:(Limón) – As Amended Ver:July 3, 2024

Urgency: No State Mandated Local Program: Yes Reimbursable: Yes

SUMMARY:

This bill temporarily requires local agencies to provide certain call-center services with workers employed in California and expands this requirement for certain call centers operated by state agencies. The bill also prohibits certain state and local call centers from using artificial intelligence (AI) or automated decision systems (ADS), if the use of AI or ADS would eliminate or automate the core job function of a worker, with some exceptions.

Specifically, this bill, until July 1, 2030, requires local agencies authorized to provide or enter into contracts relating to public benefit programs funded in any part by state funding to provide call-center services or enter contracts with call-center providers only with workers employed in California. The bill further expands existing law requiring state agencies to use California employees to operate call centers for certain public benefit programs to all programs funded in any part by state funding.

The bill further prohibits a state agency or local agency authorized to provide or enter into contracts relating to public benefit programs funded in any part by state funding to provide call center services related to public benefits from using artificial intelligence (AI) or automated decision systems (ADS) that eliminates or automates the core job function of a worker, with certain exceptions. If a state agency or local agency uses or enter into a contract that uses AI or ADS that impacts the core job function of a worker, this bill requires the agency to notify workers, their collective bargaining unions, and the public 60 days in advance with a description and purpose of the use of AI or ADS. The bill excludes emergency services from its provisions, including 988 and 911, among other exceptions.

This bill does not apply to contracts or subcontracts entered into before January 1, 2025.

FISCAL EFFECT:

1. Unknown, ongoing local costs, until July 1, 2030, to local agencies to terminate existing out-of-state contracts, procure new contracts with call centers staffed by workers employed in California, or modify existing contracts that currently utilize AI or ADS. If the Commission on State Mandates determines the bill’s requirements to be a reimbursable state mandate, the state would need to provide General Fund to reimburse these costs to local agencies.
2. Ongoing General Fund costs to the California Department of Social Services (CDSS), of an unknown amount, potentially in the tens of millions of dollars annually, until July 1, 2030, to expand current restrictions on out-of-state call centers to all public-benefit programs. The full extent of out-of-state call centers used by CDSS is unknown. However, CDSS DSS anticipates additional General Fund costs of $11.7 million annually to apply this bill to the Summer-Electronic Benefit Transfer program, known as SUN Bucks. These increased costs are attributable to the current differential for an in-state technical assistance call center, which is $4.50 per client.
3. Ongoing General Fund costs, of about $120,000 to the Department of Healthcare Services and its contractor to determine whether the use of AI or ADS is impacting the core job function of its workers.

According to the Legislative Analyst’s Office, the General Fund faces a structural deficit in the tens of billions of dollars over the next several fiscal years.

COMMENTS:

1. Purpose. According to the author:

The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving the most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not artificial intelligence.

1. Background. Current law requires a state agency authorized to provide or enter into a contract relating to public benefit programs funded by state funding to provide call-center services or enter contracts with call-center providers only with workers employed in California. Current law defines “public benefit program” to mean California Work Opportunity and Responsibility to Kids, CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.

This bill, until July 1, 2030, applies this requirement to local agencies. The bill also expands the definition of public benefit program to include programs funded in any part by state funding. The bill applies this new definition to both state and local agencies.

Current law does not address the use AI or ADS with regard to call-centers operated by public agencies. This bill defines AI as “an engineered or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.” The bill defines ADS as “a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking.”

AI and ADS are growing in use and popularly in the public and private sectors. In the public sector, the technologies have led to efficiencies in delivery of services in some cases. For example, AI is particularly apt at fraud detection due to its ability to rapidly and accurately analyze large datasets. However, the technologies have also led to pitfalls. For example, AI may contain biases and these biases may result in discriminatory outcomes for certain groups. In addition, some sectors have experienced job losses as a result of AI and ADS performing duties previously performed by a worker.

This bill seeks to regulate the use of AI and ADS at public agency call centers, with the objective of both avoiding some of the documented pitfalls of AI and ADS and shielding public sector jobs from replacement by AI or ADS.

1. Support and Opposition. This bill is co-sponsored by the California Labor Federation, AFL-CIO and California State Council of the Service Employees International Union and supported by various other groups. The California State Council of the Service Employees International Union writes:

The growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. Without worker engagement implementation can be disastrous, with worker engagement even disastrous implementations can overtime serve the purpose needed. That distinction is one that this legislation underscores.

The Association of California Healthcare Districts and California Association of Public Hospitals and Health Systems have submitted a letter of concern and seek exemptions from the bill’s provisions. They write:

Public healthcare systems and healthcare districts are local public agencies and serve a large portion of the state’s Medi-Cal population. To meet the needs of their patients and continue delivering high quality care to millions of patients each year, these entities operate a number of different types of call centers access their systems…we are concerned about the about the broad scope of SB 1220 and its potential to disrupt the appropriate and effective use of technology and call centers in our systems.

Analysis Prepared by: Natasha Collins / APPR. / (916) 319-2081

Date of Hearing: July 2, 2024

ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION

Rebecca Bauer-Kahan, Chair

SBPCA Bill Id:SB 1220

Author:(Limón) – As Amended Ver:May 16, 2024

AS PROPOSED TO BE AMENDED

SENATE VOTE: 32-7

SUBJECT: Public benefits contracts: phone operator jobs

SYNOPSIS

With the arrival of the latest generation of large language models such as ChatGPT-4, chatbots have infiltrated virtually every white collar professional setting. Such models can process enormous datasets, identify hidden patterns, make decisions with a high degree of accuracy and reliability, and converse fluidly with humans. The widespread adoption of this technology, which may continue to develop at an explosive pace, carries the potential to disrupt jobs in virtually every sector of the economy. Goldman Sachs recently predicted that roughly two-thirds of U.S. occupations could see a quarter to half of their workload displaced by artificial intelligence (AI).

Especially vulnerable are call center jobs, which can offer decent salaries, commission, and benefits, and which often do not require a college degree—providing a path to upward mobility for individuals with disadvantaged backgrounds. While AI currently falls well short of the emotional and intellectual complexity that these services frequently demand, AI can adequately handle many ordinary customer service problems at a fraction of the cost of employing a human. For many companies, the potential diminishment in quality of customer service is worth the savings.

This is not the first time call centers have faced the threat of outsourcing: when the Schwarzenegger and Davis Administrations approved out-of-country vendor contracts for CalFRESH call centers, California enacted protections against outsourcing for call center workers who provide services related to public benefits. As the executive branch has begun procuring generative AI tools for various functions, including call center services for tax guidance, this bill seeks to extend those protections to address the threat of automation by AI.

As proposed to be amended, the bill would, among other things, extend to local governments the existing prohibition on outsourcing public contracts for call services related to public benefits. The bill would also prohibit state and local agencies from contracting for or using call center services related to public benefits that use AI or automated decisionmaking systems (ADS) that eliminate or automate workers’ core job functions, with certain exceptions. Before an agency may use or contract for the use of an AI or ADS that impacts the core job function of a worker, the agency must first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system. The bill’s provisions sunset July 1, 2030.

The bill is sponsored by the California Labor Federation and the California State Council of the Service Employees International Union. The bill is supported by the California Democratic Party, the California School Employees Association, Orange County Employees Association, and TechEquity Action. The bill is opposed by various 211 service providers and certain local chapters of United Way. Amendments set forth below appear to address opponents’ concerns.

The bill previously passed the Assembly Committee On Public Employment and Retirement on a 5-0 vote.

SUMMARY: Extends to local governments the prohibition on outsourcing public contracts for call services related to public benefits. Prohibits state and local agencies from contracting for or using call center services related to public benefits that use AI or ADS that eliminate or automate workers’ core job functions, with certain exceptions. Requires, before an agency may use or contract for the use of AI or ADS that impacts the core job function of a worker, that the agency first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system. Specifically, this bill:

1. Extends to local governments the existing requirement that, for state agency contracts for call center services related to public benefits programs, the contractor certify in their bid that the services will be performed solely by workers employed in California.
2. Prohibits state and local agencies from contracting for or using call center services related to public benefits that use AI or ADS that eliminate or automate the core job function of a worker. A noncompliant contractor may be penalized for the amount the contractor or subcontractor paid for the cost of the AI or ADS to perform core job functions that were eliminated or automated.
3. Requires such agencies, before using or contracting for the use of AI or ADS that impact the core job function of a worker, to first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system.
4. Exempts from the notice requirement, the following uses of AI or ADS:
   • Phone trees or prompts that route callers to appropriate staff, information, or options.
   • Tools that support workers by providing suggestions, AI-powered searches and support of policy, procedural, or training information.
   • AI used by workers to assist callers with public benefits or assist workers in service delivery.
5. Defines, among other terms:
   • “Core job function” as a component of call service delivery that requires the exercise of judgment and discretion to successfully accomplish a complex task.
   • “Public benefits program” as California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, Covered California, Employment Development Department Insurance, Child Welfare Services, California Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), and other public benefits provided by a local government agency funded in any part by state funding from the California Health and Human Services Agency to provide the same or similar benefits.
6. Exempts:
   • Emergency services such as 988 and 911.
   • Call center services provided for a local public benefit program in response to an emergency or disaster, including mutual aid, or temporary staff relocations during the emergency or disaster, and until normal operations have been restored.
   • A contract or subcontract by a county human services agency for translation or interpretation services.
7. Provides that it does not apply to contracts related to a request for proposal or bid that occurred before January 1, 2025.
8. Sunsets July 1, 2030.

EXISTING LAW:

1. Requires that a state agency authorized to enter into contracts relating to public benefits programs only contract for services provided by a call center that directly serve applicants for, recipients of, or enrollees in, those public benefit programs may contract only with a contractor that certifies in its bid for the contract that the services provided under the contract and any subcontract performed under the contract will be performed soley with workers employed in California. A knowing violation of this provision is punishable by a civil penalty of up to $10,000 in an action brought by a public prosecutor. (Pub. Cont. Code § 12140(a)(1).)
2. Defines:
   1. “Call center” as a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.
   2. “Public benefit programs” as California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System. (Id. at (b).)
3. Requires that a contract under 1) provides that in the event a contractor or subcontractor performs the contract or the subcontract for call center services with workers not employed in California during the life of the contract, the state has the right to terminate the contract for noncompliance and the contractor or subcontractor must pay a penalty to the state agency in an amount equal to the amount paid by the state agency for the percentage of work that was performed. (Id. at (c).)
4. Provides that a state agency may conduct a solicitation that does not apply 1) if the California Health and Human Services Agency or the board of the California Health Benefit Exchange makes a specified determination. (Id. at (d).)
5. Provides that the requirements described above do not apply to the following:
   1. A contract or subcontract, if implementation would violate the specific terms of the Agreement on Government Procurement of the World Trade Organization or any other bilateral or regional free trade agreement to which the State of California has consented.
   2. A contract or subcontract, including extensions of that contract or subcontract, that results from a request for proposal or bid that occurred prior to the effective date of the act that adds this chapter.
   3. A contract for a public benefit program between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care, and any subcontract performed under that contract, or a disability insurer or specialized health insurer regulated by the Department of Insurance, and any subcontract performed under that contract. (Id. at (e).)

FISCAL EFFECT: As currently in print, this bill is keyed fiscal.

COMMENTS:

1) Author’s statement. According to the author, “The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving the most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not artificial intelligence.”

2) AI. In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.

ADS. In addition to AI, this bill addresses “automated decisionmaking systems” which generally refer to systems that use AI to replace or supplement human decisionmaking ability. ADS are used to decide whom to hire, what treatments patients should receive, whether to grant parole, and many other life-altering decisions that used to be made entirely by human beings.

GenAI. AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.

The creation of text, imagery, video, and audio by GenAI has the potential to change the world by automating repetitive tasks and fostering creativity. But care should be taken to avoid what New York University Professor Meredith Broussard describes as “the notion that computers are more ‘objective’ or ‘unbiased’ because they distill questions and answers down to mathematical evaluation…an unwavering faith that if the world used more computers, and used them properly, social problems would disappear and we’d create a digitally enabled utopia.”^[1] Many aspects of the real world are not reducible to data, and many complex situations present competing demands that human beings remain best equipped to assess. Examples of general challenges with AI follow.

Bias and discrimination. Bias and discrimination can result from skewed training data. There is a famous saying in computer science: “garbage in, garbage out.” The performance of an AI is directly impacted by the quality, quantity, and relevance of the data used to train it.^[2] If the data used to train the AI is biased, the tool’s outputs will be similarly biased. Rather than eliminating human bias, AI can reinforce and mask it with the guise of mathematical neutrality.

Interpretability. Issues such as those described above can be masked by the inscrutability of the most advanced AI models. AI that use “deep learning”—machine learning that uses multilayer (deep) neural networks to analyze vast datasets—can infer intricate patterns and relationships from complicated data, but exactly how they do so is not always obvious. Deep neural networks are sometimes criticized for being “black boxes” that generate predictions and outcomes that cannot be clearly explained.^[3] “It’s often observed in the field that the most powerful models are on the whole the least intelligible, and the most intelligible are among the least accurate.”^[4] If a decision-making process cannot be explained and clearly understood, how can it be guaranteed that the decision reached is fair and free of bias? Research into the creation of “explainable” AI is ongoing.

Hallucinations: When an AI produces a result that is not grounded in reality, it is said to be “hallucinating.” Text-generators such as ChatGPT do not fundamentally understand the text they are producing. They calculate one word or symbol at a time – if they estimate that the next word/symbol in a sentence should be a period, then the sentence ends. Otherwise, the sentence continues. It is a testament to the ingenious architecture of these systems that their outputs are remotely coherent. But while the text they produce is cogent, it is not always correct.

“These systems live in a world of language,” said Melanie Mitchell, an A.I. researcher at the Santa Fe Institute. “That world gives them some clues about what is true and what is not true, but the language they learn from is not grounded in reality. They do not necessarily know if what they are generating is true or false.”^[5]

Privacy. Just as humans cannot intentionally forget information they have learned, it is not currently possible to remove data from a trained AI. Unlike an Excel spreadsheet, which stores data in neat columns, AI stores data in the connections between “neurons” in a “neural network.” Each of these connections is influenced by every piece of training data, and a large model like ChatGPT-4 is reported to have more than 1.7 trillion connections. It is not possible to specifically alter these connections in order to remove data without fundamentally changing the model; as a result, for data to be removed, the model must be retrained from scratch. ChatGPT-4 is estimated to have taken 4-7 months to train.

What happens when an AI is trained on extremely sensitive information – for example, an individual’s DNA sequence, or their social security number, or their intimate photos, or their immigration status? The same thing that happens when an AI is trained on any other type of information: the AI digests it, and then retains it forever. AI are fundamentally different from other forms of data storage. They are black holes in the information ecosystem, with “training” acting as their event horizons. Once data has crossed this threshold it cannot be removed.

3) Automation of jobs. The net effect that the widespread adoption of AI – especially GenAI – will have on labor is unknown. In 2020 the World Economic Forum published a report suggesting that 97 million new jobs may be created, while 85 million jobs may be displaced.^[6] However, the report goes on to state that though job creation currently outpaces job destruction, the rate of creation is slowing while the rate of destruction continues to accelerate.

Furthermore, quantifying only job creation and destruction fails to fully capture AI’s effect on the labor economy. A huge number of jobs are exposed to AI in one way or another, and these positions may fundamentally change as AI becomes more common. A recent International Monetary Fund analysis found that nearly 40 percent of global employment is exposed to AI.^[7] Goldman Sachs recently predicted that roughly two-thirds of U.S. occupations could see a quarter to half of their workload displaced by AI.^[8]

Technological advances regularly transform the labor economies of societies; the development of GenAI differs from historical advances, however, in that it is likely to predominantly affect skilled workers. A 2023 Pew Research Report describes this phenomenon:

Consider customer service agents. Evidence shows that AI could either replace them with more powerful chatbots or it could enhance their productivity. AI may also create new types of jobs for more skilled workers – much as the internet age generated new classes of jobs such as web developers. Another way AI-related developments might increase employment levels is by giving a boost to the economy by elevating productivity and creating more jobs overall. Overall, AI is designed to mimic cognitive functions, and it is likely that higher-paying, white-collar jobs will see a fair amount of exposure to the technology.^[9]

However, this report does not account for the effect that autonomous machines may have on jobs involving mechanical or physical tasks. One job that could be seriously affected by the adoption of autonomous vehicles is long-haul trucking:

A natural division of labor in trucking might be that advanced autonomous trucks drive themselves over the long haul, and humans take the wheel for the endpoints—what’s often called the “last mile” in transportation and logistics. In 2017, Uber announced such an approach: an autonomous truck network, connected by local hubs throughout the country. Autonomous trucks would run the long hauls between the hubs, and human truckers would pilot the trucks from hubs to delivery.^[10] 

Creative jobs may be similarly at risk, as GenAI allows scripts, voices, images, and videos to be generated automatically. In 2023, the Writers Guild of America (WGA) and Screen Actors Guild-American Federation of Television and Radio Artists (SAG-AFTRA) both engaged in prolonged strikes, partially driven by concerns that AI would displace jobs in their various industries.^[11]

Much depends on whether the breakneck rate of AI progress continues. A decade ago, deep learning models could barely distinguish dogs from cats. Four years ago, large language models could barely produce sentences at the level of a preschooler. This year, GPT-4 passed the bar exam.^[12] Current chatbots readily pass for educated adults and can serve as useful assistants. Several factors could cause the vertiginous pace of algorithmic progress to plateau. But if trends continue, AI models could jump from human-level intelligence to superintelligence this decade. Nearly every cognitive job—analyzing legislation included—could be susceptible to automation.^[13]

4) GenAI Executive Order and the tax call center “sandbox.” In September of 2023, Governor Gavin Newsom signed Executive Order N-12-23 with the stated goals of “deploy[ing] GenAI ethically and responsibly throughout state government, protect[ing] and prepar[ing] for potential harms, and remain[ing] the world’s AI leader.” Among other things, the Executive Order directs agencies and departments to issue guidelines for the procurement, uses, and training for application of GenAI. By March 2024, the California Department of Technology was required to “establish the infrastructure to conduct pilots of GenAI projects, including California Department of Technology approved environments, or ‘sandboxes,’ to test such projects. The Executive Order also requires agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state.^[14]

A November 2023 report from the Government Operations Agency stated that “GenAI has the potential to dramatically improve service delivery outcomes and increase access to and utilization of government programs.” ^[15] The report noted numerous specific potential public sector uses of GenAI that could benefit Californians. The report found that chatbots can be used “to guide users through services in their preferred language,” “[i]ncrease first-call resolution for state service centers,” “[r]educe call wait and handle time at state customer service centers,” and that GenAI can recommend and explain government services and program eligibility to constituents.^[16] “The report cautions that generative models can produce convincing but inaccurate results, deliver different answers to the same prompt, and suffer model collapse, when predictions stray away from accurate results. Generative AI also carries the risk of automation bias, when people become overly trusting and reliant on automated decision-making.”^[17]

On January 16, 2024, a “request for innovative ideas” was issued for California Department of Tax and Fee Administration (CDFTA) “seeking a solution that will use GenAI to swiftly search expansive reference materials and provide possible responses to taxpayers who reach out via telephone and live chat.”^[18] The request noted that CDFTA’s Customer Service Center fielded 661,155 calls in fiscal year 2022/23. The service provided by CDFTA “ranges from answering general questions regarding CDTFA tax and fee programs to in-depth analysis and guidance for customers to help them file accurate and timely returns, make online payments, register for CDTFA programs, and navigate other compliance-related services.” The goal of the request was to solicit proposals to “utilize generative artificial intelligence (GenAI) to search reference materials and the CDFTA website to provide possible responses to taxpayer inquiries via calls, chatbot, live chat, and emails.” This would “minimize the risk of incorrect information being provided to taxpayers, reduce the time needed to research responses, decrease wait times for taxpayers, and provide tools to improve monitoring of operations.”^[19]

On May 9, 2024, GovOps announced the state’s six-month partnership with five companies to evaluate GenAI tools in various “sandbox” testing environments, including CDTFA’s call center, which will pair with vendors IgnyteGroup and SymSoft Solutions, LLC. At the end of the six-month trial, the Office of Data and Innovation will assess whether the state will purchase any of the tools.^[20]

5) Purpose. This bill seeks to protect public benefits call center workers and ensure the quality of the service they provide. The bill does so by prohibiting state and local agencies from outsourcing public benefit call center jobs to contractors that use AI or ADS to automate the core job functions of public benefit call center workers. Except in specified use cases, before AI or ADS can be used in a way that impacts the core job function of workers, the agency must first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system.

6) AI and call centers. Call center jobs, which can offer decent salaries, commission and benefits, and often do not require a college degree, provide a path to upward mobility for individuals with disadvantaged backgrounds.^[21] When the Schwarzenegger and Davis Administrations approved out-of-country vendor contracts for CalFRESH call centers, California enacted protections against outsourcing for call center workers who provide services related to public benefits. (AB 2508, Bonilla, Ch. 824, Stats. 2021.) These jobs again face the threat of outsourcing—this time, to AI.

In the private sector, call centers are rapidly incorporating AI. According to a recent Washington Post article, 80% of 300 IT and call center leaders surveyed indicated that conversational AI products are essential for their companies, with more than half reporting they had already invested in the technology.^[22] The British telecommunications giant BT recently announced plans to cut around 55,000 jobs, up to 20% of which would be to customer services as staff are replaced by AI.^[23] In another survey of 2000 US call center workers, a majority reported their employers were automating some of their work and “[n]early two-thirds of respondents said they felt it was somewhat or very likely that increased use of bots would lead to layoffs within the next two years.”^[24]

The impact to workers could be devastating: US call center jobs often offer decent salaries, commission and benefits, and often do not require a college degree.^[25] This can provide a path to upward mobility for individuals with disadvantaged backgrounds. As the authors and sponsors note in background materials provided to the Committee: “Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of need, ranging from global pandemics to intense natural disasters, public benefit call center workers have stood with people in need of guidance. By contracting with AI, the state is spending money to push public servants onto public benefits.”

Automating call center services can also harm consumers. In some cases, AI has been deployed in call center services to disastrous effect. As described in the Assembly Committee On Public Employment and Retirement’s analysis of this bill:

For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named “Tessa.” The version of Tessa that was initially tested and studied was a rule-based chatbot; meaning, it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “[we] were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI; meaning, it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began informing callers to avoid “unhealthy foods” and to only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders or certain complicating health factors. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400 percent interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93 percent of the AI’s fraud charges were wrong, however much of the damage had already been done.

The author and sponsors also argue that human call center workers can empathize with callers and provide personalized support that an AI system cannot replicate. This is particularly important for ensuring that Californians who are vulnerable and in immediate need of public safety net services get the service they deserve.

There is a middle way: While some call centers have fully automated many services, others are attempting to use it in a way that supplements rather than supplants. “AI software tools help call center workers do their jobs better: transcription software analyzes call conversations in real-time and provides agents with ideal solutions; other tools identify angry customers and provides techniques to calm them down.” Such support can allow workers to focus on more complex tasks and on connecting empathically with callers, potentially leading to higher customer satisfaction and lower worker turnover.^[26] AI can also expedite service delivery; many state customer service hotlines have long wait-times that could be cut down by integration of AI.

The author and sponsors assert that this bill does not preclude AI from assisting workers with their job tasks; rather, it “clarifies that a worker must be a human being and not artificial intelligence.”

7) This bill. Under existing law, if a state agency contracts for call center services related to public benefits programs, the contractor must certify in their bid that the services will be performed solely by workers employed in California. As proposed to be amended, this bill extends that requirement to local governments. The bill also prohibits state and local agencies from contracting for or using call center services related to “public benefits”—defined as several enumerated programs, including CalWORKs, CalFresh, Medi-Cal, and other programs funded in part by state funding from the California Health and Human Services Agency—that use AI or ADS that eliminate or automate workers’ “core job functions,” defined under the bill as “a component of call service delivery that requires the exercise of judgment and discretion to successfully accomplish a complex task.”

Before an agency may use or contract for the use of AI or ADS that impacts the core job function of a worker, the agency must first notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system. However, the notice requirement does not apply to uses of AI or ADS for the following:

• • Phone trees or prompts that route callers to appropriate staff, information, or options.
  • Tools that support workers by providing suggestions, AI-powered searches and support of policy, procedural, or training information.
  • AI used by workers to assist callers with public benefits or assist workers in service delivery.

The bill exempts from its scope emergency services such as 988 and 911, services provided for a local public benefit program in response to an emergency or disaster, and contracts by a county human services agency for translation or interpretation services. The bill also provides that it does not apply to contracts related to a request for proposal or bid that occurred before January 1, 2025. The bill sunsets July 1, 2030.

8) Amendments. In response to various concerns, the author and sponsors have worked with the Committee and stakeholders to make a number of changes to narrow and refine the scope of the bill and address certain opposition concerns. In broad strokes, the amendments:

• Remove impact assessments. Amendments remove this requirement and instead require agencies to notify employees, bargaining unit representatives, and the public of planned uses AI or ADS tech that impacts their core job functions. This reduces the burden on public agencies while providing necessary information to workers related to potential AI-related impacts to their jobs.
• Redefine “core job function.” Amendments define this concept in terms of the skills required to complete a component of call service delivery—the exercise of judgment and discretion to successfully accomplish a complex task—rather than in terms of the ability of the call center worker.
• Clarify the penalty for contractors or subcontractors who eliminate or automate core job functions in violation of the bill. Under existing law—which prohibits outsourcing call center jobs to non-Californians—if a call center’s contractor outsources jobs, the agency for whom the contractor provides the service may cancel the contract and recoup the money that went to outsourced jobs. The bill in print incorporates AI-replaced jobs into this provision but the penalty is still based on the amount paid to workers outside California. Amendments clarify that the agency can recover whatever was paid for AI services that replaced core job functions.
• Exempt:
  • Emergency services: Amendments remove 911 and 988 from the bill’s ambit. This is consistent with the bill’s focus on the provision of public benefits. Emergency services workers are subject to extensive training requirements; while it seems unlikely that AI will supplant workers in this space, it is important to retain the flexibility to incorporate AI as needed to ensure efficiency in service delivery.
  • Local emergencies: The bill in print provides emergency exemptions for state agencies. At the request of the County Welfare Directors Association (CWDA), amendments extend this to local governments.
  • Translation services contracted by counties: This amendment is also being made at the request of CWDA, in light of their existing national contract for translation services enabling translation into the five most spoken languages within their jurisdiction.
  • State contracts with health plans and insurers: Amendments restore an existing exemption for contracts managed by Department of Managed Health Care or Department of Insurance. This likely resolves concerns raised by health plans and insurers.
• Remove express reference to 211 services. Providers of the 211 hotline serve as a hub for health, human, and social services. In response to requests from 211 providers, the amendments remove express reference to 211. Such services are not mandated; the providers worry that singling out 211 could give an unfair advantage to competitors that use AI using a different hotline number. While the bill would still apply to all such referral services, the removal of this reference puts all of these providers on the same footing.
• Sunset the bill July 1, 2030: This will enable the Legislature to revisit this issue in light of technological developments between now and the end of this decade to determine whether to extend or replace the bill’s provisions.
• Make other technical and clarifying changes.

The amendments are as follows:

12140. (a) (1) Notwithstanding any other law, any state agency authorized to provide or enter into contracts relating to public benefit programs, and any local government agency authorized to provide or enter into contracts relating to public benefit programs funded in any part by state funding, shall only provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California.

(2) With regard to contracts specified in paragraph (1), a contractor shall certify in its bid that any services provided by the contractor or its subcontractors are to be performed with and by workers employed in California. A certification required by this paragraph is not made under penalty of perjury. Any contractor that knowingly provides false information in the certification required by this paragraph shall be subject to a civil penalty in an amount of up to ten thousand dollars ($10,000), in addition to any other civil remedies available to the state agency. An action for a civil penalty under this paragraph may be brought by any public prosecutor in the name of the people of the State of California.

(3) A state agency, or a local agency authorized to provide or enter into contracts relating to public benefit programs funded in any part by state funding to provide call center services related to public benefits, 211 services, or 988 services, as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code, shall not use artificial intelligence (AI) or automated decision systems (ADS) that eliminate or automate the core job function of a worker. That agency shall not contract with a party to provide call center services related to public benefit programs that utilizes AI or ADS to eliminate or automate the core job function of a worker.

(4) If a state agency, or local agency authorized to provide or enter into contracts relating to public benefit programs funded in any part by state funding to provide services related to public benefits, 211 services, or 988 services, as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code uses AI or ADS that impact the core job function of a worker, or contracts with a party to provide call center services related to public benefits utilizing AI or ADS that impact the core job function of a worker, then that agency shall notify the workers, their collective bargaining representatives, and the public 60 days in advance with a general description of the AI or ADS system and the intended purpose for the AI or ADS system.comply with all of the following requirements:

(A) Provide access to a human call center worker upon request of a public caller.

(B) (i) Develop an impact assessment report on the AI or ADS proposed to be used in the call center. That report shall include all of the following elements:

(I) A detailed description of the technology and all its intended uses and roles in providing call center services.

(II) A description of the data used by the technology, including the specific categories of data that will be processed as input and any data used to train the model that the technology relies upon.

(III) A description of the types of outputs produced by the technology.

(IV) Potential impacts on current or future employees, including, but not limited to, job loss, hour reductions, and material changes in core job duties.

(ii) The agency shall notify employees potentially affected by the workplace technology, and their collective bargaining representative, at least 60 days before drafting the impact assessment report, as specified in clause (i).

(iii) The agency shall conduct all major phases of the impact assessment report with the direct, meaningful, and sustained involvement of employees impacted and their collective bargaining representative, if applicable. Sixty days before the ratification of the contract, the agency shall provide the report to employees potentially affected by the workplace technology, or their collective bargaining representative.

(5) For purposes of this section, AI or ADS may be used as a tool to support call center workers to provide services. Permissive uses of limited AI that do not impact the core job function of a worker and are not subject to the notification requirement under paragraph (4) include all of the following:

(A) Phone trees or prompts that that filter route callers to appropriate staff, information, or optionsdepartments or language audios.

(B) Tools that support workers by providing suggestions, AI-powered searches and support of policy, procedural, or training information. Screening tools to provide initial suggestions for workers.

(C) AI used by workers to assist callers with public benefits or assist workers in service delivery.

(b) For purposes of this section:

(1) “Artificial intelligence,” or “AI,” means an engineered or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.

(2) “Automated decision systems,” or “ADS” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons.

(3) “Call center” means a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.

(4) “Core job function” means a component of call service delivery that requires the exercise of judgment and discretion to successfully accomplish a complex task. ability of a call center worker to fully exercise judgment throughout the service delivery process.

(5) “Impact assessment report” means a report by an agency on AI or ADS used in the provision of call center services for public benefit programs by the agency or a contractor and its impact on the core job function of call center workers.

(5) (A) “Public benefit programs” means California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, Covered California, Employment Development Department Insurance, Child Welfare Services, California Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), and other public benefits provided by a local government agency funded in any part by state funding from the California Health and Human Services Agency to provide the same or similar benefits as those described in this section.

(B) “Public benefit programs” do not include emergency services such as 988 and 911.

(7) “State agency” means a state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code.

(8) “Worker” means a natural person.

(c) (1) The contract shall provide that in the event a contractor or subcontractor performs the contract or the subcontract for call center services without call center workers responding to calls and are employed in California during the life of the contract, or who eliminates or automates or the core job function of workers through AI or ADS technology as described in paragraph (3) of subdivision (a), the state or local government agency has the right to terminate the contract for noncompliance and the contractor or subcontractor shall pay a penalty to the state agency or local government agency in an amount equal to the amount paid by the state agency or local government agency for the percentage of work that was performed with workers not employed in California, or the amount the contractor or subcontractor paid for the cost of AI or ADS to perform core job functions that were eliminated or automated by the AI or ADS.

(2) The penalty authorized in paragraph (1) shall be in addition to any other applicable penalty, including, but not limited to, the penalty provided in paragraph (2) of subdivision (a).

(d) (1) Notwithstanding subdivision (a), a state agency may conduct a solicitation without applying this section if the California Health and Human Services Agency or the board of the California Health Benefit Exchange makes any of the following determinations:

(A) A prior solicitation was conducted and the bids received were priced unreasonably high as a result of including these provisions.

(B) A prior solicitation was conducted and no bids were received as a result of including these provisions.

(C) The services are needed in cases of emergency where immediate acquisition is necessary for the protection of the public health, welfare, or safety.

(2) If the agency or board makes a determination described in paragraph (1), that entity shall submit a report to the Assembly Committee on Jobs, Economic Development and the Economy and to the Senate Committee on Labor and Industrial Relations on or before the last day of the quarter following the quarter in which the determination was made. The report shall include the reason for making the determination.

(e) This section shall not apply to the following:

(1) A contract or subcontract, if implementation would violate the specific terms of the Agreement on Government Procurement of the World Trade Organization or any other bilateral or regional free trade agreement to which the State of California has consented.

(2) A contract or subcontract, including extensions of that contract or subcontract, that results from a request for proposal or bid that occurred before the effective date of the act that adds this chapter.

(3) A contract for a public benefit program between a state or local agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care, and any subcontract performed under that contract, or a disability insurer or specialized health insurer regulated by the Department of Insurance, and any subcontract performed under that contract.

(4) Emergency services such as 988 and 911.

(5) Call center services provided for a local public benefit program in response to an emergency or disaster, including mutual aid, or temporary staff relocations during the emergency or disaster, and until normal operations have been restored. This shall include non-emergency or non-disaster services that have been disrupted by the emergency or disaster.

(6) A contract or subcontract by a county human services agency for translation or interpretation services.

(f) Notwithstanding any other law, this section shall apply to call center services for public benefit programs regardless of the state or local agency or contractor that administers the program or provides the public benefit.

(g) This section shall be construed so as to not conflict with, and be applied consistent with, federal law.

(h) The amendments made to this section by the act adding this subdivision shall not apply for contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.

(i) The amendments to this section made by Senate Bill No. 1220 (2023-24 Reg. Sess.) repeal July 1, 2030.

[ . . .]

9) Related legislation. SB 1446 (Smallwood-Cuevas, 2024) would limit the use of self-service checkout stations in grocery retail stores and retail drug establishments and requires advance notification of the implementation of consequential workplace technology to workers, their representatives, and the public. The bill will be heard in this Committee in the same hearing as this bill.

AB 2508 (Bonilla, Ch. 824, Stats. 2012) enacted protections against outsourcing for call center workers who provide services related to public benefits.

ARGUMENTS IN SUPPORT: SEIU California, a co-sponsor of the bill, writes:

While AI and automated systems can be very helpful in helping streamlining workload and assisting public employees in serving higher caseloads than they could on their own it is the duty of the State to regulate the use and purpose of technology. Especially in highly sensitive settings, such as safety net services where the public are already in an extremely vulnerable position.

AI and automated systems run on often unknown algorithms fed with data that may be unreliable or replicate unwanted interactions. This creates a level of risk that can be particularly harmful to individuals in dire need of benefits and services like unemployment or mental health services.

The growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. Without worker engagement implementation can be disastrous, with worker engagement even disastrous implementations can overtime serve the purpose needed. That distinction is one that this legislation underscores.

For instance, the inaugural role out of CalSAWs only had limited worker engagement, in its initial implementation it significantly increased workload, delayed benefits for recipients, putting counties and the most vulnerable populations into crisis. Workers have since been included in every transition and have helped turn the use of CalSAWs into a tool that is now being utilized by almost every county. Our caseloads to worker ratio in almost every single agency are well above recommended guidelines, the use and role of technology exists but that value added is only when the workers are at the helm.

California Labor Federation, the bill’s other co-sponsor, adds:

When an individual calls state or local agencies for support, they need a person who can empathize with and who can understand their situation. Individuals do not need a lifeless voice or chat box providing automated decisions without effective solutions. Many callers are reluctant to talk about their personal or financial struggles and the job of call center workers is to make callers comfortable and read their verbal cues to elicit information.

Ultimately, AI simply cannot provide the level of empathy and effectiveness provided by a person. Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of need, ranging from global pandemics to intense natural disasters, public benefit call center workers have stood with people in need of guidance. Without protections from AI, call center workers face the erosion and eventual elimination of their jobs. By contracting with AI, the State is spending money to push public servants onto public benefits.

To protect workers and the quality of public services, SB 1220 will prohibit state and local governments from outsourcing public benefit call center jobs to firms or employers using artificial intelligence or automated decision systems (ADS) to automate the core job functions of public benefit call center workers. [. . .]

ARGUMENTS IN OPPOSITION: Numerous 211 providers oppose the bill unless it is amended to remove reference to 211—a change that the author has agreed to, as described above. Additionally, the trade associations for health plans and insurers oppose the bill unless it is amended to restore an exemption under existing law for contracts with health care services plans regulated the Department of Managed Health Care results. The author has also agreed to make this change, as described above.

REGISTERED SUPPORT / OPPOSITION:

Support

California Labor Federation, Afl-cio (co-sponsor)
California State Council of Service Employees International Union (co-sponsor)

American Federation of State, County and Municipal Employees, Afl-cio
California Democratic Party

California School Employees Association
Orange County Employees Association
TechEquity Collaborative

Opposition

Orange County United Way
United Way of The Wine Country

Oppose Unless Amended

2-1-1 Humboldt Info and Resource Ctr
211 LA County
211 San Diego
Association of California Life and Health Insurance
California Association of Health Plans
Community Action Partnership of Kern
Contra Costa Crisis Center
Inland Southern California United Way
United Way of Northern California
United Way of Stanislaus County

Analysis Prepared by: Josh Tosney / P. & C.P. / (916) 319-2200

1. Broussard, “Artificial Unintelligence,” MIT Press, 2019. ↑

2. Rohit Sehgal, “AI Needs Data More Than Data Needs AI”, Forbes, Oct. 5, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/10/05/ai-needs-data-more-than-data-needs-ai/. ↑

3. Neil Savage, “Breaking into the black box of artificial intelligence,” Nature, March 29, 2022. ↑

4. Brian Christian, “The Alignment Problem: Machine Learning and Human Values” (Norton 2020, First Ed.), p. 85. ↑

5. Cade Metz, “What Makes A.I. Chatbots Go Wrong?” New York Times, March 29, 2023, www.nytimes.com/2023/03/29/technology/ai-chatbots-hallucinations.html. ↑

6. World Economic Forum. "The Future of Jobs Report 2020," October, 2020. ↑

7. Kristalina Georgieva, "AI Will Transform the Global Economy. Let’s Make Sure It Benefits Humanity," International Monetary Fund, January 14, 2024, www.imf.org/en/Blogs/Articles/2024/01/14/ai-will-transform-the-global-economy-lets-make-sure-it-benefits-humanity. ↑

8. Goldman Sachs, “Generative AI could raise global GDP by 7%” (Apr. 5, 2024), https://www.goldmansachs.com/intelligence/pages/generative-ai-could-raise-global-gdp-by-7-percent.html. ↑

9. Rakesh Kochhar, "Which U.S. Workers Are More Exposed to AI on Their Jobs?," Pew Research Center, July 26, 2023, www.pewresearch.org/social-trends/2023/07/26/which-u-s-workers-are-more-exposed-to-ai-on-their-jobs/. ↑

10. Karen Levy, “Robo Truckers and the AI-Fueled Future of Transport,” Wired, December 6, 2022, www.wired.com/story/autonomous-vehicles-transportation-truckers-employment/. ↑

11. Ryan Broderick, "AI can’t replace humans yet — but if the WGA writers don’t win, it might not matter," Polygon, May 31, 2023, www.polygon.com/23742770/ai-writers-strike-chat-gpt-explained. ↑

12. Pablo Arredondo, GPT-4 Passes the Bar Exam: What That Means for Artificial Intelligence Tools in the Legal Profession (Apr. 19, 2023), https://law.stanford.edu/2023/04/19/gpt-4-passes-the-bar-exam-what-that-means-for-artificial-intelligence-tools-in-the-legal-industry/. ↑

13. See generally Leopold Aschenbrenner, “Situational Awareness: The Decade Ahead” (Jun. 2024), https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf. ↑

14. Executive Department—State of California, Executive Order N-12-23 (Sept. 6, 2023), https://www.gov.ca.gov/wp-content/uploads/2023/09/AI-EO-No.12-_-GGN-Signed.pdf ↑

15. Cal GovOps, “State of California: Benefits and Risks of Generative Artificial Intelligence Report” (Nov. 2023), p. 2, https://www.govops.ca.gov/wp-content/uploads/sites/11/2023/11/GenAI-EO-1-Report_FINAL.pdf. ↑

16. Id. at p. 12. ↑

17. Khari Johnson, “California plans to use AI to answer your tax questions” (Feb. 8, 2024) CalMatters, https://calmatters.org/economy/technology/2024/02/cdtfa-generative-ai/. ↑

18. CDTFA, “Leveraging GenAI to Enhance Services for Taxpayers” (Jan. 16, 2024), https://www.cdtfa.ca.gov/news/24-02.htm. ↑

19. “RFI2 - Request for Innovative Ideas - GenAI Solution for Call Center Team Productivity” (Jan. 16, 2024), https://caleprocure.ca.gov/event/77601/0000029440. ↑

20. “California Signs Paternships to Utilize GenAI (May 9, 2024), https://calmatters.org/wp-content/uploads/2024/05/GenAI-awards-press-release_Final_5.9.2024_.docx-GenAI-awards-press-release_Final_5.9.2024_.docx.pdf. ↑

21. Id. ↑

22. Pranshu Verma, “ChatGPT provided better customer service than his staff. He fired them” (Oct. 3, 2023) The Washington Post, https://www.washingtonpost.com/technology/2023/10/03/ai-customer-service-jobs/. ↑

23. Tom Espiner, “BT to cut 55,000 jobs with up to a fifth replaced by AI” (May 18, 2023) BBC, https://www.bbc.com/news/business-65631168. ↑

24. Emma Goldberg, “‘Training My Replacement’: Inside a Call Center Worker’s Battle with A.I.” (Jul. 19, 2023) New York Times, https://www.nytimes.com/2023/07/19/business/call-center-workers-battle-with-ai.html?smid=nytcore-ios-share&referringSource=articleShare.

    https://www.nytimes.com/2023/07/19/business/call-center-workers-battle-with-ai.html?smid=nytcore-ios-share&referringSource=articleShare ↑

25. Id. ↑

26. Pranshu Verma, “ChatGPT provided better customer service than his staff. He fired them” (Oct. 3, 2023) The Washington Post, https://www.washingtonpost.com/technology/2023/10/03/ai-customer-service-jobs/. ↑

Date of Hearing: June 19, 2024

ASSEMBLY COMMITTEE ON PUBLIC EMPLOYMENT AND RETIREMENT

Tina S. McKinnor, Chair

SBPCA Bill Id:SB 1220

Author:(Limón) – As Amended Ver:May 16, 2024

SENATE VOTE: 32-7

SUBJECT: Public benefits contracts: phone operator jobs

SUMMARY: Prohibits the use of Artificial Intelligence (AI) or Automated Decision Systems (ADS) by a state agency authorized to provide or enter into contracts relating to public benefit programs, as defined, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, among other provisions. Specifically, this bill:

1. Prohibits a state agency, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using AI or ADS that eliminate or automate the core job functions of a worker.
2. Provides that if a state, or local agency funded in any part by state funding to provide services related to public benefits or services, uses AI or ADS that impact the core job function of a worker, then that agency must comply with all of the following requirements:
   1. Provide access to a human call center worker upon request of a public caller.
   2. Develop an impact assessment report on the AI or ADS to be used in the call center, and the report must the following:
      1. A detailed description of the technology and all its intended uses and roles in providing call center services;
      2. A description of the data used by the technology, including the specific categories of data that will be processed as input and any data used to train the model that the technology relies upon;
      3. A description of the types of outputs produced by the technology; and,
      4. Potential impacts on current or future employees, including, but not limited to, job loss, hour reductions, and material changes in core job duties.
3. Requires a state agency, if a state agency is authorized to provide or enter into contracts relating to public benefit programs, as defined, 211 services, or 988 services, and any local government agency funded in any part by state funding to provide services related to public benefits or services, to only provide services through a call center that directly serves callers with services performed solely with and by workers employed in California, as specified.
4. Requires the agency to notify employees potentially affected by the workplace technology, and their collective bargaining representative, at least 60 days before drafting the impact assessment.
5. Requires the agency to conduct all major phases of the impact assessment report with the direct, meaningful, and sustained involvement of employees impacted and their collective bargaining representative, if applicable. Sixty days before the ratification of the contract, the agency shall provide the report to employees potentially affected by the workplace technology, or their collective bargaining representative.
6. Extends the right to terminate a contract to a local government agency in the event a contractor or subcontractor performs the contract without workers or workers not employed in California, as specified.
7. Deletes a current exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
8. Provides that the provisions of this bill shall not apply for contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.
9. Defines “core job function” to mean the ability of a call center worker to fully exercise judgement throughout the service delivery process. This does not include:
   1. Phone trees that filter callers to appropriate departments or language audios.
   2. Screening tools to provide initial suggestions on eligibility.
   3. AI or digital tech used by workers to assist callers with public benefits.
10. Defines “impact assessment report” to mean a report by a contractor on AI or ADS technologies used in the provisions of call center services and its impact on call center workers.
11. Adds Covered California, Employment Development Insurance and other public benefits provided by a local government agency funded in any part by state funding to the definition of “public benefits programs.”
12. Defines “artificial intelligence,” or “AI,” to mean an engineering or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
13. Defines “automated decision systems,” or “ADS” to mean a computational process derived from machine learning, statistical modeling, data analytics or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons.
14. Defines “state agency” to mean:
    1. A state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services as described in Article 6.3 (commencing with Section 53123.1 of Chapter 1 of part 1 of Division 2 of Title 5 of the Government Code.).
15. Defines “worker” to mean a person.

EXISTING LAW:

1. Authorizes, a state agency authorized to enter into contracts relating to public benefit programs, as defined, but limits such contracts to for services by a call center that directly serves applicants or recipients for, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed will be performed solely with workers employed in California, ^[1] and among other provisions:
   1. Defines “call center” to mean a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.^[2]
   2. Defines “public benefit programs,” for the above provisions to mean California Work Opportunity and Responsibility to Kids (CalWORKS), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.^[3]
   3. Provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.^[4]
   4. Authorizing the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.^[5]
2. Defines “agency” or “state agency” as any department, board, office, authority, commission, or other agency of state government.^[6]

FISCAL EFFECT: According to the Senate Appropriations Committee, this bill would result in the following:

• Unknown, potentially significant ongoing reimbursable mandate costs for local agencies to terminate existing out-of-state contracts, procure new contracts with call centers that are staffed solely by workers employed in California, or modify existing contracts that currently utilize AI or ADS. (General Fund).
• Unknown fiscal impact to the Department of Social Services (DSS) and the Department of Health Care Services (DHCS). To the extent that DSS and DHCS utilize AI or ADS that impact or replace the core job functions of a worker in the administration of their various programs, then they will likely incur costs to modify existing contracts or IT systems, hire additional staff, or revise processes.
• The Department of General Services, the California Department of Insurance, the Department of Managed Health Care, and the Office of Emergency Services do not anticipate a fiscal impact.

COMMENTS:

1. Dual Referral

The committee is informed that this bill also is referred to the Assembly Committee on Privacy and Consumer Protection, which has and retains primary jurisdiction relating to this subject and this bill.

1. Background: What is Artificial Intelligence (AI)?

In its analysis of this bill, the Senate Committee on Governmental Organization states that, “AI refers to the development of computer systems that can perform tasks typically requiring human intelligence. These tasks include learning, reasoning, problem-solving, perception, and natural language understanding. AI enables computers to learn from data and experiences, allowing them to handle complex tasks without explicit programming. AI has the potential to revolutionize numerous sectors, including healthcare, finance, and transportation, by improving efficiency, reducing costs, and enhancing decision-making capabilities.

“One of the most advanced and well-known AI systems, GPT-4, was developed by OpenAI and released on Pi Day (3/14), 2023. GPT-4 has demonstrated remarkable capabilities compared to its predecessor, GPT-3.5, which had been released to the world just six months prior. For example, GPT-4 scored around the top 10 percent on a simulated bar exam, while GPT-3.5 scored around the bottom 10 percent.

“GPT-4 also scored a five out of five on several Advanced Placement (AP) exams, including Art History, Biology, Environmental Science, Macro- and Microeconomics, and Psychology, showcasing a knowledge across a diverse range of subjects. GPT-4 is particularly impressive at reasoning, scoring in the 93^rd percentile on the SAT Evidence-Based Reading & Writing exam and in the 99^th percentile on the Graduate Record Examination (GRE) Verbal exam.

“The remarkable advancements in AI, exemplified by GPT-4's impressive performance on various professional and academic benchmarks, showcase its potential for practical applications in diverse fields. From law and education to science and technology, AI's ability to understand complex tasks and generate nuanced responses could lead to transformative solutions and new insights across multiple disciplines.

There are different types of AI, such as rule-based systems to make decisions or carry out tasks, and machine learning systems, which learn from data and experience to improve their performance over time. Deep learning, a type of machine learning, utilizes neural networks that mimic the structure and function of the human brain. Neural networks are used in deep learning to process large amounts of data and learn to recognize patterns, make predictions, and perform other tasks with a high degree of accuracy.

“AI is already being used in a wide range of applications, including virtual assistants, image and speech recognition, fraud detection, predictive analytics, and autonomous vehicles. AI excels at fraud detection and predictive analytics due to its ability to rapidly and accurately analyze large datasets, detect patterns and anomalies challenging for humans to discern, and learn from historical data to predict future events. By detecting potential fraudulent activity and predicting future outcomes, AI can help businesses and governments make more informed decisions, save time and resources, and prevent financial losses.

“While AI has the potential to revolutionize government services and decision-making, it also presents several challenges and concerns. One major issue is the potential for bias in AI systems. AI algorithms are only as unbiased as the data they are trained on, and if that data is biased or incomplete, it can result in discriminatory outcomes. For example, if an AI system is used to assess job applicants, it may inadvertently discriminate against certain groups if the data used to train the system is biased towards certain demographics. This could perpetuate existing inequalities in the workplace. To minimize bias in AI systems, it is crucial to ensure that the data used to train these systems is representative of the target population.

“Another concern is the potential for privacy violations. AI systems can collect and process vast amounts of personal data, and if not properly secured, that data could be vulnerable to misuse or hacking. This is particularly concerning in government contexts, where sensitive information such as criminal records, medical histories, and financial information is often collected and stored. Additionally, there is concern that AI could displace jobs in certain sectors, particularly those that rely heavily on repetitive or routine tasks. While AI has the potential to automate many tasks and increase efficiency, it could also lead to significant job losses and create new inequalities in the workforce.

“Further, there is the risk of unintended consequences. AI systems are complex and difficult to fully understand, and it is possible that they could produce unexpected or unintended outcomes. This is particularly concerning in government contexts where decisions made by AI systems could have far-reaching consequences for individuals and communities. For example, if an AI system is used to determine eligibility for social services, errors or biases in the system could result in individuals being denied important benefits. As such, it is important that AI is developed and implemented in a responsible and ethical manner, with proper safeguards and oversight in place to mitigate these risks.”

1. AI Currently Used in California Government Services

Information in the analysis of this bill by the Senate Committee on Governmental Organization, “AI is already being used in various ways within California government, including in the areas of transportation, healthcare, public safety, natural resource management, and emergency preparation and response. For instance, in 2020, the University of California, San Diego (UCSD) developed an AI-powered tool that can analyze chest X-rays to detect potential cases of COVID-19. This tool helped healthcare professionals quickly identify patients who needed immediate care, leading to more efficient allocation of medical resources during the pandemic.

“In 2019, the number of traffic fatalities in the United States was 36,096, which is more than 11.17 deaths per 100,000 population. These numbers underscore the critical need to improve traffic, pedestrian, and bicyclist safety. The AI Pedestrian Traffic Safety System – tested on 12 hours of traffic videos captured from existing traffic cameras in the city of Los Angeles – demonstrated more than 98% accuracy in counting pedestrians in real time. This project occurred as part of LA’s public commitment to the global initiative Vision Zero, which endeavors to reduce severe injuries and deaths in roadway collisions. Los Angeles’s specific Vision Zero goal is to eliminate all traffic deaths and serious injuries by 2025. At full scale, the system can cover all 56,000 miles of LA streets and in real time.

“After the 2018 Camp Fire in California, AI technology was used to analyze satellite imagery and assess the extent of the damage caused by the fire. This information helped emergency responders prioritize their efforts, allocate resources more efficiently, and speed up the recovery process for affected communities.

“However, it is important to acknowledge that there are potential risks and negative consequences associated with AI. Bias, privacy violations, job displacement, and unintended consequences are some of the concerns associated with AI. These issues must be addressed and mitigated to ensure that the benefits of AI are maximized while minimizing any potential risks.

“In 2016, a study by ProPublica revealed that a widely used AI-based risk assessment tool, COMPAS, was biased against African-American defendants in predicting their likelihood of reoffending. The algorithm was found to be more likely to falsely flag black defendants as high-risk and white defendants as low-risk, perpetuating existing racial biases in the criminal justice system.

1. Skynet: Ground Zero? When Use of AI Goes Wrong.

The author’s office points to a number of examples where AI or automation have replaced human workers and the results have been devastating. For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named “Tessa.” The version of Tessa that was initially tested and studied was a rule-based chatbot; meaning, it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “[we] were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI; meaning, it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began informing callers to avoid “unhealthy foods” and to only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders or certain complicating health factors. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400 percent interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93 percent of the AI’s fraud charges were wrong, however much of the damage had already been done.

Although the information that the author presents above is not nearly as vastly encompassing as the reaches of Skynet, the aforementioned cautionary examples serve as the fundamental rationale for this bill relative to governmental use of AI.^[7]

1. AI and ADS: Another Form of Contracting Out?

The committee is reminded of its informational hearing on May 10, 2023, titled “Strengthening California through the Public Sector and Its Workforce,” which focused on various subjects, including contracting out of public employee work.

During that hearing, the committee heard from a number of panelists, including experts from the University of California at Berkeley Labor Center, where data was provided and substantial concerns were expressed by other panelists about the ongoing and increasing reliance by public employers, including the state – as an employer, on a contingent, part-time, temporary, contracted out, or retired annuitant workforce to fill public sector vacancies, or to perform the duties of willing and capable existing and prospective public employees. These concerns also detailed how public employers are increasingly relying on these forms of employment and in a manner that has deleterious effects on wage growth, employee morale, employer-employee relations, and the need to ensure operational consistency and quality in the provision of services to the public that could be performed by permanent employees.

These concerns also detailed how public employers are increasingly relying on these forms of employment and in a manner that has deleterious effects on wage growth, employee morale, employer-employee relations, and the need to ensure operational consistency and quality in the provision of services to the public that could be performed by permanent employees.

Following the committee’s hearing last year, on April 17, 2024, the committee held another information relating to that subject titled: “Public Service Delivery and Workforce Wellbeing – Addressing the Vacancy Crisis in Local Government,” where civil service vacancies, impacts of vacancies on the civil service workforce and services, and collaborative solutions to address civil service vacancies, were discussed.

Although AI and ADS was not discussed during either of those hearings, AI and ADS may be helpful or useful in certain circumstances. However, its use may be viewed as a surreptitious form of state and local government outsourcing of employment relating to public benefits services, particularly relating to call centers for such programs and services, and in a manner that may lead to workforce reductions or job elimination that may negatively impact the quality and provision of certain services for public benefit programs. This bill resolves to address those concerns.

1. Statement by the Author

For these purposes, the author states that, “[this bill] will prohibit the state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or other technology that completely replaces workers,” and “… defines a worker as a person.” Further, “[it] does not preclude AI from assisting workers with their job tasks; clarifies that a worker must be a human being and not AI; and, ensures that there will be human assistance available when people call to receive help with public benefit programs, serving most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but… clarifies that a worker must be a human being and not AI.”

1. Comments by Supporters

Generally, supporters of this bill express that, while AI may be helpful in some circumstances, it is an algorithm devoid of human understanding, which can have disastrous effects on the public regarding programs and services, especially for those who are most in need. Further, to protect the jobs of workers and ensure quality in the provision of public services, [this bill] prohibits state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or ADS to automate the core job functions of public benefit call center workers. While not prohibiting the use of AI, [this bill] will require agencies to produce impact assessment reports when utilizing AI, ADS, or similar technology to guarantee that workers are at the center of the decision-making process during potential AI deployment.

1. Comments by Opponents

Generally, opponents express that this bill amends existing law to add “211” services – a comprehensive information and referral service that can only be provided by one entity per county, which must receive authorization from the Public Utilities Commission for use of the 211 dialing code. They further express that expanding the scope of existing law to cover this service risks jeopardizing service funding resulting in diminished community service, and because counties are neither mandate nor funded by the state to provide this service, they are partners in the service. By placing a very small number of 211 providers in a competitive disadvantage while burdening them with costly restrictions, this bill may result in diminished services for nearly 2 million Californians who annually rely on the service.

As such, several opponents request an amendment to remove reference to 211 service(s) from this bill.

1. Prior or Related Legislation

SB 892 (Padilla, 2024) requires CDT to develop and adopt regulations to create an AI risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an ADS unless the contract includes a clause that provides a complete risk assessment, as specified. (Pending in the Senate Governmental Organization Committee)

SB 893 (Padilla, 2024) requires the Government Operations Agency, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California Artificial Intelligence Research Hub to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI Research and development that seeks to harness the technology’s full potential for public benefit while safeguarding privacy, advancing security, and addressing risks and potential harms to society, as specific. (Pending in the Senate Governmental Organization Committee)

SB 896 (Dodd, 2024) codifies much of EO N-12-23, ensuring state agencies advance safeguards and consumer protections around AI including notification requirements and the evaluation of automated decision-making systems being utilized by a state agency or department for risk potential prior to adoption. (Pending in the Senate Governmental Organization Comitee)

AB 2286 (Aguiar-Curry, 2024) prohibits the operation of an autonomous vehicle with a gross vehicle weight of 10,001 pounds or more on public roads for test purposes, transporting goods, or transporting passengers without a human safety operator physically present in the autonomous vehicle at the time of operation. (Pending in the Assembly Transportation Committee)

AB 2370 (Cervantes, 2024) prohibits AI from being used to replace community college faculty for purposes of providing academic instruction to, and to regular interaction with, students in a course instruction, and would authorize AI to only be used as a peripheral tool to support faculty in carrying out those tasks for uses such as course development, assessment, and tutoring. (Pending in the Assembly Privacy and Consumer Protection Committee)

SB 313 (Dodd, 2023) would have established the Office of AI, within the California Department of Technology, with the power and authority necessary to guide the design, use, and deployment of automated systems by a state agency, as specified. This bill was held in the Senate Appropriations Committee.

AB 316 (Aguiar-Curry, 2023) would have restricted an autonomous vehicle with a gross vehicle weight of 10,001 pounds or more from being operated on public roads for testing purposes, transportation goods, or transporting passengers without a human safety operator physically present in the autonomous vehicle at the time of operation. This bill was voted by the Governor stating that:

“Assembly Bill 316 is unnecessary for the regulation and oversight of heavy-duty autonomous vehicle technology in California, as existing law provides sufficient authority to create the appropriate regulatory framework. In 2012, the California Legislature provided the Department of Motor Vehicles (DMV) with the authority to regulate the testing and deployment of autonomous vehicles on public roads in California. As part of its oversight and regulatory responsibilities, DMV consults with the California Highway Patrol, the National Highway Traffic Safety Administration, and others with relevant expertise to determine the regulations necessary for the safe operation of autonomous vehicles on public roads.

“DMV continuously monitors the testing and operations of autonomous vehicles on California roads and has the authority to suspend or revoke permits as necessary to protect the public's safety. Autonomous vehicle technology is evolving and DMV remains committed to keeping our rules up to date to reflect its continued development in California. DMV held public workshops with interested stakeholders earlier this year to inform the development of future rulemakings for both light-duty and heavy-duty autonomous vehicles. This rulemaking will be a transparent, public process where subject matter experts and other stakeholders will have the opportunity to shape the regulations related to the safe operations of autonomous vehicles in California. The draft regulations are expected to be released for public comment in the coming months.

“In addition to safety, my Administration has long been concerned with the impact of technology on the future of work - so much so that in 2019 we convened, with participation from a variety of organized labor leaders including the Teamsters, UFCW, and SEIU, a robust Future of Work Task Force. That effort led to the publication of a report that guides our work on issues of emerging technology and its impacts on California's workforce. But our efforts don't end there. I am committed to incentivizing career pathways and training for the necessary workforce specifically associated with this technology. As such, I am directing the Labor and Workforce Development Agency to lead a stakeholder process next year to review and develop recommendations to mitigate the potential employment impact of testing and deployment of autonomous heavy-duty vehicles.

“Considering the longstanding commitment of my Administration to addressing the present and future challenges for work and workers in California, and the existing regulatory framework that presently and sufficiently governs this particular technology, this bill is not needed at this time. For these reasons, I cannot sign this bill. My Administration remains open to working with the author, sponsors, and other stakeholders on the right approach to safely test and deploy this evolving technology in California, while also addressing and mitigating any potential impacts to jobs.”

Chapter 824, Statutes of 2012 (AB 2508, Bonilla) prohibits a state agency authorized to contract for public benefits programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in those programs, and that the work will be performed solely by workers employed in California.

REGISTERED SUPPORT / OPPOSITION:

Support

California Labor Federation, AFL-CIO (Co-sponsor)

California State Council of the Service Employees International Union (Co-sponsor)

California Democratic Party

California School Employees Association, AFL-CIO

Orange County Employees Association

Opposition

2-1-1 Humboldt Information and Resource Center (Oppose Unless Amended)

211 California (Oppose Unless Amended)

211 San Diego (Oppose Unless Amended)

CommUnify (formerly Community Action Commission of Santa Barbara County) (Oppose

Unless Amended)

Eden I&R, Inc. / 211 Alameda County (Oppose Unless Amended)

Family Resource Center (Oppose Unless Amended)

Inland Southern California United Way (Oppose Unless Amended)

Interface Children & Family Services (Oppose Unless Amended)

Orange County United Way (Oppose Unless Amended)

United Way Bay Area (Oppose Unless Amended)

United Way Monterey County (Oppose Unless Amended)

United Way of Northern California (Oppose Unless Amended)

United Way of Stanislaus County (Oppose Unless Amended)

United Ways of California (Oppose Unless Amended)

Analysis Prepared by: Michael Bolden / P. E. & R. / (916) 319-3957

1. Section 12140 (a) of the Public Contract Code (PCC). ↑

2. Section 12140 (b) (1) of the PCC. ↑

3. Section 12140 (b) (2), ibid. ↑

4. Section 12410 (e), inclusive, of the PCC. ↑

5. Section 12140 (c) (1) of the PCC. ↑

6. Section 19050.9 of the Government Code. ↑

7. Based on the “Terminator” film franchise, the fictional Skynet is an artificial neural network-based conscious group mind and artificial general superintelligence system created for Strategic Air Command-North American Aerospace Defense Command by Cyberdyne Systems. ↑

 THIRD READING  

Bill No: SB 1220

Author: Limón (D)

Amended: 5/16/24

Vote: 21

SENATE GOVERNMENTAL ORG. COMMITTEE: 10-3, 4/9/24

AYES: Dodd, Wilk, Alvarado-Gil, Archuleta, Bradford, Glazer, Padilla, Portantino, Rubio, Smallwood-Cuevas

NOES: Jones, Ochoa Bogh, Seyarto

NO VOTE RECORDED: Ashby, Nguyen, Roth

SENATE LOCAL GOVERNMENT COMMITTEE: 5-1, 4/17/24

AYES: Durazo, Glazer, Skinner, Wahab, Wiener

NOES: Seyarto

NO VOTE RECORDED: Dahle

SENATE APPROPRIATIONS COMMITTEE: 5-2, 5/16/24

AYES: Caballero, Ashby, Becker, Bradford, Wahab

NOES: Jones, Seyarto

SUBJECT: Public benefits contracts: phone operator jobs

SOURCE: California Labor Federation

Service Employees International Union

DIGEST: This bill prohibits a state agency authorized to provide or enter into contracts relating to public benefit programs, as defined, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using Artificial Intelligence (AI), or Automated Decision Systems (ADS) that eliminate or automate the core job functions of a worker. If those state or local agencies use AI or ADS that impacts the core functions of a worker, the bill requires state and local agencies to comply with certain requirements, including notification to employees and the development of an impact assessment report, as specified.

ANALYSIS:

Existing law:

1. Prohibits, with specified exemptions, a state agency authorized to enter into contracts relating to public benefit programs, as defined, from contracting for services provided by a call center that directly serves applicants for, recipients for, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed will be performed solely with workers employed in California.
2. Defines “public benefit programs,” for the above provisions to mean California Work Opportunity and Responsibility to Kids (CalWORKS), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.
3. Defines “call center” for the above provisions to mean a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.
4. Provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
5. Authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill:

1. Prohibits a state agency, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using AI, or ADS that eliminate or automate the core job functions of a worker.
2. Provides that if a state, or local agency funded in any part by state funding to provide services related to public benefits or services, uses AI or ADS that impact the core job function of a worker, then that agency shall comply with all of the following requirements:
   1. Provide access to a human call center worker upon request of a public caller.
   2. Develop an impact assessment report on the AI or ADS to be used in the call center. That report shall included all of the following elements:
      1. A detailed description of the technology and all its intended uses and roles in providing call center services.
      2. A description of the data used by the technology, including the specific categories of data that will be processed as input and any data used to train the model that the technology relies upon.
      3. A description of the types of outputs produced by the technology.
      4. Potential impacts on current or future employees, including, but not limited to, job loss, hour reductions, and material changes in core job duties.
3. Requires a state agency, if a state agency is authorized to provide or enter into contracts relating to public benefit programs, as defined, 211 services, or 988 services, and any local government agency funded in any part by state funding to provide services related to public benefits or services, to only provide services through a call center that directly serves callers with services performed solely with and by workers employed in California, as specified.
4. Requires the agency to notify employees potentially affected by the workplace technology, and their collective bargaining representative, at least 60 days before drafting the impact assessment.
5. Requires the agency to conduct all major phases of the impact assessment report with the direct, meaningful, and sustained involvement of employees impacted and their collective bargaining representative, if applicable. Sixty days before the ratification of the contract, the agency shall provide the report to employees potentially affected by the workplace technology, or their collective bargaining representative.
6. Extends the right to terminate a contract to a local government agency in the event a contractor or subcontractor performs the contract without workers or workers not employed in California, as specified.
7. Deletes a current exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
8. Provides that the provisions of this bill shall not apply for contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.

Background

Author’s Statement. According to the author’s office, “SB 1220 will prohibit the state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or other technology that completely replaces workers. This bill defines a worker as a person. It does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI. The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI.”

Artificial Intelligence. AI refers to the development of computer systems that can perform tasks typically requiring human intelligence. These tasks include learning, reasoning, problem-solving, perception, and natural language understanding. AI enables computers to learn from data and experiences, allowing them to handle complex tasks without explicit programming. AI has the potential to revolutionize numerous sectors, including healthcare, finance, and transportation, by improving efficiency, reducing costs, and enhancing decision-making capabilities.

One of the most advanced and well-known AI systems, GPT-4, was developed by OpenAI and released on Pi Day (3/14), 2023. GPT-4 has demonstrated remarkable capabilities compared to its predecessor, GPT-3.5, which had been released to the world just six months prior. For example, GPT-4 scored around the top 10% on a simulated bar exam, while GPT-3.5 scored around the bottom 10%.

GPT-4 also scored a five out of five on several Advanced Placement (AP) exams, including Art History, Biology, Environmental Science, Macro- and Microeconomics, and Psychology, showcasing a knowledge across a diverse range of subjects. GPT-4 is particularly impressive at reasoning, scoring in the 93^rd percentile on the SAT Evidence-Based Reading & Writing exam and in the 99^th percentile on the Graduate Record Examination (GRE) Verbal exam.

The remarkable advancements in AI, exemplified by GPT-4's impressive performance on various professional and academic benchmarks, showcase its potential for practical applications in diverse fields. From law and education to science and technology, AI's ability to understand complex tasks and generate nuanced responses could lead to transformative solutions and new insights across multiple disciplines.

There are different types of AI, such as rule-based systems to make decisions or carry out tasks, and machine learning systems, which learn from data and experience to improve their performance over time. Deep learning, a type of machine learning, utilizes neural networks that mimic the structure and function of the human brain. Neural networks are used in deep learning to process large amounts of data and learn to recognize patterns, make predictions, and perform other tasks with a high degree of accuracy.

AI is already being used in a wide range of applications, including virtual assistants, image and speech recognition, fraud detection, predictive analytics, and autonomous vehicles. AI excels at fraud detection and predictive analytics due to its ability to rapidly and accurately analyze large datasets, detect patterns and anomalies challenging for humans to discern, and learn from historical data to predict future events. By detecting potential fraudulent activity and predicting future outcomes, AI can help businesses and governments make more informed decisions, save time and resources, and prevent financial losses.

While AI has the potential to revolutionize government services and decision-making, it also presents several challenges and concerns. One major issue is the potential for bias in AI systems. AI algorithms are only as unbiased as the data they are trained on, and if that data is biased or incomplete, it can result in discriminatory outcomes. For example, if an AI system is used to assess job applicants, it may inadvertently discriminate against certain groups if the data used to train the system is biased towards certain demographics. This could perpetuate existing inequalities in the workplace. To minimize bias in AI systems, it is crucial to ensure that the data used to train these systems is representative of the target population.

Another concern is the potential for privacy violations. AI systems can collect and process vast amounts of personal data, and if not properly secured, that data could be vulnerable to misuse or hacking. This is particularly concerning in government contexts, where sensitive information such as criminal records, medical histories, and financial information is often collected and stored. Additionally, there is concern that AI could displace jobs in certain sectors, particularly those that rely heavily on repetitive or routine tasks. While AI has the potential to automate many tasks and increase efficiency, it could also lead to significant job losses and create new inequalities in the workforce.

Further, there is the risk of unintended consequences. AI systems are complex and difficult to fully understand, and it is possible that they could produce unexpected or unintended outcomes. This is particularly concerning in government contexts where decisions made by AI systems could have far-reaching consequences for individuals and communities. For example, if an AI system is used to determine eligibility for social services, errors or biases in the system could result in individuals being denied important benefits. As such, it is important that AI is developed and implemented in a responsible and ethical manner, with proper safeguards and oversight in place to mitigate these risks.

AI in Government Services. AI is already being used in various ways within California government, including in the areas of transportation, healthcare, public safety, natural resource management, and emergency preparation and response. For instance, in 2020, the University of California, San Diego (UCSD) developed an AI-powered tool that can analyze chest X-rays to detect potential cases of COVID-19. This tool helped healthcare professionals quickly identify patients who needed immediate care, leading to more efficient allocation of medical resources during the pandemic.

In 2019, the number of traffic fatalities in the United States was 36,096, which is more than 11.17 deaths per 100,000 population. These numbers underscore the critical need to improve traffic, pedestrian, and bicyclist safety. The AI Pedestrian Traffic Safety System – tested on 12 hours of traffic videos captured from existing traffic cameras in the city of Los Angeles – demonstrated more than 98% accuracy in counting pedestrians in real time. This project occurred as part of LA’s public commitment to the global initiative Vision Zero, which endeavors to reduce severe injuries and deaths in roadway collisions. Los Angeles’s specific Vision Zero goal is to eliminate all traffic deaths and serious injuries by 2025. At full scale, the system can cover all 56,000 miles of LA streets and in real time.

After the 2018 Camp Fire in California, AI technology was used to analyze satellite imagery and assess the extent of the damage caused by the fire. This information helped emergency responders prioritize their efforts, allocate resources more efficiently, and speed up the recovery process for affected communities.

However, it is important to acknowledge that there are potential risks and negative consequences associated with AI. Bias, privacy violations, job displacement, and unintended consequences are some of the concerns associated with AI. These issues must be addressed and mitigated to ensure that the benefits of AI are maximized while minimizing any potential risks.

In 2016, a study by ProPublica revealed that a widely used AI-based risk assessment tool, COMPAS, was biased against African-American defendants in predicting their likelihood of reoffending. The algorithm was found to be more likely to falsely flag black defendants as high-risk and white defendants as low-risk, perpetuating existing racial biases in the criminal justice system.

When Use of AI Goes Wrong. The author’s office points to a number of examples where AI or automation have replaced human workers and the results have been devastating. For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named Tessa. The version of Tessa that was initially tested and studied was a rule-based chatbot, meaning it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “we were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI, meaning it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began to telling callers to avoid “unhealthy foods” and only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400% interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93% of the AI’s fraud charges were wrong, however much of the damage had already been done.

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes

According to the Senate Appropriations Committee, unknown potentially significant ongoing reimbursable mandate costs for local agencies to terminate existing out-of-state contracts, procure new contracts with call centers that are staffed solely by workers employed in California, or modify existing contracts that utilize AI or ADS. (General Fund)

Additionally, unknown fiscal impact to the Department of Social Services (DSS) and the Department of Health Care Services (DHCS). To the extent that DSS and DHCS utilize AI or ADS that impact or replace the core job functions of a worker in the administration of their various programs, then they will likely incur costs to modify existing contracts or IT systems, hire additional staff, or revise processes. DGS, the California Department of Insurance, the Department of Managed Health Care, and the Office of Emergency Services do not anticipate a fiscal impact.

SUPPORT: (Verified 5/16/24)

California Labor Federation (co-source)

Service Employees International Union (co-source)

California School Employees Association, AFL-CIO

Orange County Employee Association

TechEquity Collaborative

OPPOSITION: (Verified 5/16/24)

None received

ARGUMENTS IN SUPPORT: According to the California Labor Federation, “the growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. With the high volume of callers in need of support, public benefit programs such a Medi-Cal, the 988 Suicide Prevention & Mental Health Crisis Lifeline, and unemployment insurance assistance requires a support staff in a California that understands the issues facing the state’s most vulnerable populations. While not prohibiting the use of AI, SB 1220 will require agencies to produce impact assessment reports when utilizing AI, or ADS to guarantee that workers are at the center of the decision-making process during AI deployment.”

According to TechEquity Collaborative, “ultimately, AI simply cannot provide the level of empathy and effectiveness provided by a person. Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of needs, ranging from global pandemic to intense natural disasters, public benefit call center workers have stood with people in need of guidance. Without protections from AI, call center workers face the erosion and eventual elimination of their jobs.”

Prepared by: Felipe Lopez / G.O. / (916) 651-1530

5/19/24 9:00:01

**** END ****

SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION

Senator Bill Dodd

Chair

2023 - 2024 Regular

Bill No: Hearing Date: 4/9/2024

NOTE: This analysis reflects amendments that the author will be taking in committee.

SUBJECT:

DIGEST: This bill prohibits a state agency authorized to provide or enter into contracts relating to public benefit programs, as defined, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using Artificial Intelligence (AI), or Automated Decision Systems (ADS) that eliminate or automate the core job functions of a worker. If those state or local agencies use AI or ADS that impacts the core functions of a worker, the bill requires state and local agencies to comply with certain requirements, including notification to employees and the development of an impact assessment report, as specified.

ANALYSIS:

Existing law:

1. Prohibits, with specified exemptions, a state agency authorized to enter into contracts relating to public benefit programs, as defined, from contracting for services provided by a call center that directly serves applicants for, recipients for, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed will be performed solely with workers employed in California.
2. Defines “public benefit programs,” for the above provisions to mean California Work Opportunity and Responsibility to Kids (CalWORKS), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.
3. Defines “call center” for the above provisions to mean a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.
4. Provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
5. Authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill:

1. Prohibits a state agency, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using AI, or ADS that eliminate or automate the core job functions of a worker.
2. Provides that if a state, or local agency funded in any part by state funding to provide services related to public benefits or services, uses AI or ADS that impact the core job function of a worker, then that agency shall comply with all of the following requirements:
   1. Provide access to a human call center worker upon request of a public caller.
   2. Develop an impact assessment report on the AI or ADS to be used in the call center. That report shall included all of the following elements:
      1. A detailed description of the technology and all its intended uses and roles in providing call center services.
      2. A description of the data used by the technology, including the specific categories of data that will be processed as input and any data used to train the model that the technology relies upon.
      3. A description of the types of outputs produced by the technology.
      4. Potential impacts on current or future employees, including, but not limited to, job loss, hour reductions, and material changes in core job duties.
3. Requires a state agency, if a state agency is authorized to provide or enter into contracts relating to public benefit programs, as defined, 211 services, or 988 services, and any local government agency funded in any part by state funding to provide services related to public benefits or services, to only provide services through a call center that directly serves callers with services performed solely with and by workers employed in California, as specified.
4. Requires the agency to notify employees potentially affected by the workplace technology, and their collective bargaining representative, at least 60 days before drafting the impact assessment.
5. Requires the agency to conduct all major phases of the impact assessment report with the direct, meaningful, and sustained involvement of employees impacted and their collective bargaining representative, if applicable. Sixty days before the ratification of the contract, the agency shall provide the report to employees potentially affected by the workplace technology, or their collective bargaining representative.
6. Extends the right to terminate a contract to a local government agency in the event a contractor or subcontractor performs the contract without workers or workers not employed in California, as specified.
7. Deletes a current exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
8. Provides that the provisions of this bill shall not apply for contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.
9. Defines “core job function” to mean the ability of a call center worker to fully exercise judgement throughout the service delivery process. This does not include:
   1. Phone trees that filter callers to appropriate departments or language audios.
   2. Screening tools to provide initial suggestions on eligibility.
   3. AI or digital tech used by workers to assist callers with public benefits.
10. Defines “impact assessment report” to mean a report by a contractor on AI or ADS technologies used in the provisions of call center services and its impact on call center workers.
11. Adds Covered California, Employment Development Insurance and other public benefits provided by a local government agency funded in any part by state funding to the definition of “public benefits programs.”
12. Defines “artificial intelligence,” or “AI,” to mean an engineering or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
13. Defines “automated decision systems,” or “ADS” to mean a computational process derived from machine learning, statistical modeling, data analytics or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons.
14. Defines “state agency” to mean a state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services as described in Article 6.3 (commencing with Section 53123.1 of Chapter 1 of part 1 of Division 2 of Title 5 of the Government Code.)
15. Defines “worker” to mean a person.

Background

Author’s Statement. According to the author’s office, “SB 1220 will prohibit the state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or other technology that completely replaces workers. This bill defines a worker as a person. It does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI. The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI.”

Artificial Intelligence. AI refers to the development of computer systems that can perform tasks typically requiring human intelligence. These tasks include learning, reasoning, problem-solving, perception, and natural language understanding. AI enables computers to learn from data and experiences, allowing them to handle complex tasks without explicit programming. AI has the potential to revolutionize numerous sectors, including healthcare, finance, and transportation, by improving efficiency, reducing costs, and enhancing decision-making capabilities.

One of the most advanced and well-known AI systems, GPT-4, was developed by OpenAI and released on Pi Day (3/14), 2023. GPT-4 has demonstrated remarkable capabilities compared to its predecessor, GPT-3.5, which had been released to the world just six months prior. For example, GPT-4 scored around the top 10% on a simulated bar exam, while GPT-3.5 scored around the bottom 10%.

GPT-4 also scored a five out of five on several Advanced Placement (AP) exams, including Art History, Biology, Environmental Science, Macro- and Microeconomics, and Psychology, showcasing a knowledge across a diverse range of subjects. GPT-4 is particularly impressive at reasoning, scoring in the 93^rd percentile on the SAT Evidence-Based Reading & Writing exam and in the 99^th percentile on the Graduate Record Examination (GRE) Verbal exam.

The remarkable advancements in AI, exemplified by GPT-4's impressive performance on various professional and academic benchmarks, showcase its potential for practical applications in diverse fields. From law and education to science and technology, AI's ability to understand complex tasks and generate nuanced responses could lead to transformative solutions and new insights across multiple disciplines.

There are different types of AI, such as rule-based systems to make decisions or carry out tasks, and machine learning systems, which learn from data and experience to improve their performance over time. Deep learning, a type of machine learning, utilizes neural networks that mimic the structure and function of the human brain. Neural networks are used in deep learning to process large amounts of data and learn to recognize patterns, make predictions, and perform other tasks with a high degree of accuracy.

AI is already being used in a wide range of applications, including virtual assistants, image and speech recognition, fraud detection, predictive analytics, and autonomous vehicles. AI excels at fraud detection and predictive analytics due to its ability to rapidly and accurately analyze large datasets, detect patterns and anomalies challenging for humans to discern, and learn from historical data to predict future events. By detecting potential fraudulent activity and predicting future outcomes, AI can help businesses and governments make more informed decisions, save time and resources, and prevent financial losses.

While AI has the potential to revolutionize government services and decision-making, it also presents several challenges and concerns. One major issue is the potential for bias in AI systems. AI algorithms are only as unbiased as the data they are trained on, and if that data is biased or incomplete, it can result in discriminatory outcomes. For example, if an AI system is used to assess job applicants, it may inadvertently discriminate against certain groups if the data used to train the system is biased towards certain demographics. This could perpetuate existing inequalities in the workplace. To minimize bias in AI systems, it is crucial to ensure that the data used to train these systems is representative of the target population.

Another concern is the potential for privacy violations. AI systems can collect and process vast amounts of personal data, and if not properly secured, that data could be vulnerable to misuse or hacking. This is particularly concerning in government contexts, where sensitive information such as criminal records, medical histories, and financial information is often collected and stored. Additionally, there is concern that AI could displace jobs in certain sectors, particularly those that rely heavily on repetitive or routine tasks. While AI has the potential to automate many tasks and increase efficiency, it could also lead to significant job losses and create new inequalities in the workforce.

Further, there is the risk of unintended consequences. AI systems are complex and difficult to fully understand, and it is possible that they could produce unexpected or unintended outcomes. This is particularly concerning in government contexts where decisions made by AI systems could have far-reaching consequences for individuals and communities. For example, if an AI system is used to determine eligibility for social services, errors or biases in the system could result in individuals being denied important benefits. As such, it is important that AI is developed and implemented in a responsible and ethical manner, with proper safeguards and oversight in place to mitigate these risks.

AI in Government Services. AI is already being used in various ways within California government, including in the areas of transportation, healthcare, public safety, natural resource management, and emergency preparation and response. For instance, in 2020, the University of California, San Diego (UCSD) developed an AI-powered tool that can analyze chest X-rays to detect potential cases of COVID-19. This tool helped healthcare professionals quickly identify patients who needed immediate care, leading to more efficient allocation of medical resources during the pandemic.

In 2019, the number of traffic fatalities in the United States was 36,096, which is more than 11.17 deaths per 100,000 population. These numbers underscore the critical need to improve traffic, pedestrian, and bicyclist safety. The AI Pedestrian Traffic Safety System – tested on 12 hours of traffic videos captured from existing traffic cameras in the city of Los Angeles – demonstrated more than 98% accuracy in counting pedestrians in real time. This project occurred as part of LA’s public commitment to the global initiative Vision Zero, which endeavors to reduce severe injuries and deaths in roadway collisions. Los Angeles’s specific Vision Zero goal is to eliminate all traffic deaths and serious injuries by 2025. At full scale, the system can cover all 56,000 miles of LA streets and in real time.

After the 2018 Camp Fire in California, AI technology was used to analyze satellite imagery and assess the extent of the damage caused by the fire. This information helped emergency responders prioritize their efforts, allocate resources more efficiently, and speed up the recovery process for affected communities.

However, it is important to acknowledge that there are potential risks and negative consequences associated with AI. Bias, privacy violations, job displacement, and unintended consequences are some of the concerns associated with AI. These issues must be addressed and mitigated to ensure that the benefits of AI are maximized while minimizing any potential risks.

In 2016, a study by ProPublica revealed that a widely used AI-based risk assessment tool, COMPAS, was biased against African-American defendants in predicting their likelihood of reoffending. The algorithm was found to be more likely to falsely flag black defendants as high-risk and white defendants as low-risk, perpetuating existing racial biases in the criminal justice system.

When Use of AI Goes Wrong. The author’s office points to a number of examples where AI or automation have replaced human workers and the results have been devastating. For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named Tessa. The version of Tessa that was initially tested and studied was a rule-based chatbot, meaning it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “we were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI, meaning it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began to telling callers to avoid “unhealthy foods” and only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400% interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93% of the AI’s fraud charges were wrong, however much of the damage had already been done.

As Proposed to be Amended. As previously noted, this analysis reflects amendments that the author has committed to accepting in committee. A below is a brief explanation of the major amendments.

Amendment 1: Replace language related to employing sufficient workers as follows:

 Employ sufficient workers to respond to callers requesting human assistance. Provide access to a human call center worker upon request of a public caller.

Amendments 2: Update definition of “public benefit programs” as follows:

“Public benefit programs” means California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, California Health Benefit Exchange, Employment Development Department insurance, and any other needs-based program or information and referral program that refers people to, or provides benefits funded in any part by, the State of California. Covered California, Employment Development Department Insurance and other public benefits provided by a local government agency funded in any part by state funding.

Amendment 3: Delete the “or any state agency that provides benefits or services to the public,” delete prevailing wage language, and delete language pertaining to employing sufficient workers in section 12140.

12140. (a) (1) Notwithstanding any other law, any state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services, as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code, or any state agency that provides benefits or services to the public, and any local government agency funded in any part by state funding to provide services related to public benefits or services, shall only provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California, who are paid at least the prevailing minimum wage in the jurisdiction in California in which they are employed. A call center, as specified in this paragraph, shall also employ sufficient workers to respond to callers requesting human assistance.

Amendment 4: Add definition of state agency to clarify that this is only for public benefit programs, 211 services, and 988 services as follows:

“state agency” means a state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code,

Amendment 5: Delete “essential job function” and replace with “core job functions” throughout the bill and define “core job function” as follows:

“Core job function” means ability of a call center worker to fully exercise judgement throughout the service delivery process. This does not include:

(A) Phone trees that filter callers to appropriate departments or language audios.

(B) Screening tools to provide initial suggestions on eligibility.

(C) AI or digital tech used by workers to assist callers with public benefits.

Amendment 6: Redefine “Artificial Intelligence” and “automated decision systems” as follows:

(1) “Artificial intelligence,” or “AI,” means a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. an engineered or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.

(2) “Automated decision systems,” or “ADS,” means a computational process, including one derived from machine learning, statistics, or other data processing or artificial intelligence techniques, that makes, or assists in making, decisions impacting workers or the public. a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons.

Amendment 7: Delete “other digital technologies” language throughout the bill to ensure that the focus of the bill is on AI and ADS.

Prior/Related Legislation

SB 892 (Padilla, 2024) requires CDT to develop and adopt regulations to create an AI risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an ADS unless the contract includes a clause that provides a complete risk assessment, as specified. (Pending in the Senate Governmental Organization Committee)

SB 893 (Padilla, 2024) requires Govops, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California Artificial Intelligence Research Hub to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI Research and development that seeks to harness the technology’s full potential for public benefit while safeguarding privacy, advancing security, and addressing risks and potential harms to society, as specific. (Pending in the Senate Governmental Organization Committee)

SB 896 (Dodd, 2024) codifies much of EO N-12-23, ensuring state agencies advance safeguards and consumer protections around AI including notification requirements and the evaluation of automated decision-making systems being utilized by a state agency or department for risk potential prior to adoption. (Pending in the Senate Governmental Organization Comitee)

AB 2286 (Aguiar-Curry, 2024) prohibits the operation of an autonomous vehicle with a gross vehicle weight of 10,001 pounds or more on public roads for test purposes, transporting goods, or transporting passengers without a human safety operator physically present in the autonomous vehicle at the time of operation. (Pending in the Assembly Transportation Committee)

AB 2370 (Cervantes, 2024) prohibits AI from being used to replace community college faculty for purposes of providing academic instruction to, and to regular interaction with, students in a course instruction, and would authorize AI to only be used as a peripheral tool to support faculty in carrying out those tasks for uses such as course development, assessment, and tutoring. (Pending in the Assembly Privacy and Consumer Protection Committee)

SB 313 (Dodd, 2023) would have established the Office of AI, within the California Department of Technology, with the power and authority necessary to guide the design, use, and deployment of automated systems by a state agency, as specified. (Held on the Senate Appropriations Committee Suspense File)

AB 316 (Aguiar-Curry, 2023) would have restricted an autonomous vehicle with a gross vehicle weight of 10,001 pounds or more from being operated on public roads for testing purposes, transportation goods, or transporting passengers without a human safety operator physically present in the autonomous vehicle at the time of operation. (Vetoed by Governor Newsom)

AB 2508 (Bonilla, Chapter 824, Statutes of 2012) prohibits a state agency authorized to contract for public benefits programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in those programs, and that the work will be performed solely by workers employed in California.

FISCAL EFFECT: Appropriation: Fiscal Com.: Local:

SUPPORT:

California Labor Federation (Co-source)

TechEquity Collaborative

OPPOSITION:

None received

ARGUMENTS IN SUPPORT: According to the California Labor Federation, “the growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. With the high volume of callers in need of support, public benefit programs such a Medi-Cal, the 988 Suicide Prevention & Mental Health Crisis Lifeline, and unemployment insurance assistance requires a support staff in a California that understands the issues facing the state’s most vulnerable populations. While not prohibiting the use of AI, SB 1220 will require agencies to produce impact assessment reports when utilizing AI, or ADS to guarantee that workers are at the center of the decision-making process during AI deployment.”

According to TechEquity Collaborative, “ultimately, AI simply cannot provide the level of empathy and effectiveness provided by a person. Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of needs, ranging from global pandemic to intense natural disasters, public benefit call center workers have stood with people in need of guidance. Without protections from AI, call center workers face the erosion and eventual elimination of their jobs.”

DUAL REFERRAL: Senate Governmental Organization Committee & Senate Local Government Committee

SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION

Senator Bill Dodd

Chair

2023 - 2024 Regular

Bill No: SB Hearing Date: 4/9/2024

SUBJECT:

DIGEST: This bill >

ANALYSIS:

Existing law:

>

This bill:

>

Background

>

Prior/Related Legislation

>

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: Yes

SUPPORT:

>

OPPOSITION:

>

ARGUMENTS IN SUPPORT: >

ARGUMENTS IN OPPOSITION: >

SENATE COMMITTEE ON

Senator Anna Caballero, Chair

2023 - 2024 Regular Session

SB 1220 (Limón) - Public benefits contracts: phone operator jobs

Bill Summary:

SB 1220 prohibits state and local agencies authorized to provide or enter into contracts relating to public benefit programs, along with 211 and 988 services, from contracting with out-of-state call centers and restricts the use of artificial intelligence (AI) and automated decision systems (ADS) that eliminate or automate the core job functions of a worker, as specified.

*********** ANALYSIS ADDENDUM – SUSPENSE FILE ***********

The following information is revised to reflect amendments

adopted by the committee on May 16, 2024

Fiscal Impact:

• Unknown, potentially significant ongoing reimbursable mandate costs for local agencies to terminate existing out-of-state contracts, procure new contracts with call centers that are staffed solely by workers employed in California, or modify existing contracts that currently utilize AI or ADS. (General Fund).
• Unknown fiscal impact to the Department of Social Services (DSS) and the Department of Health Care Services (DHCS). To the extent that DSS and DHCS utilize AI or ADS that impact or replace the core job functions of a worker in the administration of their various programs, then they will likely incur costs to modify existing contracts or IT systems, hire additional staff, or revise processes.
• The Department of General Services, the California Department of Insurance, the Department of Managed Health Care, and the Office of Emergency Services do not anticipate a fiscal impact.

Author Amendments:

• Exempt 988 and 211 from provisions prohibiting out-of-state contracting.
• Specify provisions apply to state or local agencies authorized to provide or enter into contracts relating to public benefit programs funded
• Make clarifying changes.

-- END --

SENATE COMMITTEE ON

Senator Anna Caballero, Chair

2023 - 2024 Regular Session

SB 1220 (Limón) - Public benefits contracts: phone operator jobs

Bill Summary:

SB 1220 prohibits state and local agencies authorized to provide or enter into contracts relating to public benefit programs, along with 211 and 988 services, from contracting with out-of-state call centers and restricts the use of artificial intelligence (AI) and automated decision systems (ADS) that eliminate or automate the core job functions of a worker, as specified.

Fiscal Impact:

• Unknown, likely significant ongoing reimbursable mandate costs for counties operating 211 services and other impacted local agencies to terminate existing out-of-state contracts, procure new contracts with call centers that are staffed solely by workers employed in California, or modify existing contracts that currently utilize AI or ADS. (General Fund).
• Unknown, potentially significant fiscal impact to the 988 service and 12 regional crisis call centers in California to comply with the mandates of this bill.
• Unknown fiscal impact to the Department of Social Services (DSS) and the Department of Health Care Services (DHCS). To the extent that DSS and DHCS utilize AI or ADS that impact or replace the core job functions of a worker in the administration of their various programs, then they will likely incur costs to modify existing contracts or IT systems, hire additional staff, or revise processes.
• The Department of General Services, the California Department of Insurance, the Department of Managed Health Care, and the Office of Emergency Services do not anticipate a fiscal impact.

Background:

Current law requires call centers serving people enrolled in five specific public benefit programs be staffed solely by workers employed in California. This requirement, created by AB 2508 (Bonilla, Chapter 824, Statutes of 2012) applies to:

• California Work Opportunity and Responsibility to Kids (CalWORKs)
• CalFresh
• Medi-Cal
• Healthy Families
• California Healthcare Eligibility, Enrollment, Retention System.

Current call center in-state labor restrictions do not apply to:

• Contracts or subcontracts that would violate the specific terms of the Agreement on Government Procurement of the World Trade Organization or any other bilateral or regional free trade agreement to which the State of California has consented.
• Contracts or subcontracts, including extensions, that relate to a request for proposal or bid that occurred before this bill takes effect, should it be signed into law.
• Places where it may conflict with federal law.
• Contracts for a public benefit program between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care (DMHC) and any subcontract performed under that contract, or a disability insurer or specialized health insurer regulated by the Department of Insurance (DOI), and any subcontract performed under that contract.

Proposed Law:

• Expands the prohibition against contracting with out-of-state call centers to all state agencies that provide public benefit programs, 211 services, 988 services, and any local government agency funded in any part by state funding to provide public benefits or services.
• Prohibits a state agency, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using AI, or ADS that eliminate or automate the core job functions of a worker.
• Provides that if a state, or local agency funded in any part by state funding to provide services related to public benefits or services, uses AI or ADS that impact the core job function of a worker, then that agency shall provide access to a human call center worker upon public caller request and develop an impact assessment report on the AI or ADS to be used in the call center, as specified.
  • Requires the agency to notify employees potentially affected by the workplace technology, and their collective bargaining representatives at least 60 days before drafting the impact assessment report.
  • Requires the agency to conduct all major phases of the impact assessment report with the direct and meaningful, and sustained involvement of employees impacted and their collective bargaining representative.
• Expands the definition of “public benefit program” to include Covered California, Employment Development Department Insurance, and other public benefits provided by a local government agency funded in any part by state funding.
• Extends the right to terminate a contract to a local government agency in the event a contractor or subcontractor performs the contract without workers or workers not employed in California, as specified.
• Deletes a current exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
• Provides that the provisions of this bill shall not apply for contracts or subcontracts, including extensions of contracts or subcontracts that result from a request for proposal or bid that occurred before January 1, 2025.

Staff Comments:

The 988 Lifeline is a national network of local crisis centers, and 988 has been designated as the national three-digit dialing code that routes callers to the National Suicide Prevention & Mental Health Crisis Lifeline. While the 12 regional Lifeline crisis call centers in California handle most in-state calls to 988, if one of those centers can’t answer a call it is routed to one of the designated national back-up centers. As noted in the Senate Committee on Local Government analysis of this bill, there are nearly 200 call centers nationwide, but not all serve as back-up centers. Between January 2023 and January 2024, California call centers averaged 29,030 calls per month and the in-state answer rate was 88%—the highest in-state answer rate in the country—meaning 12% of the calls were routed to out-of-state 988 operators.

Under this bill, California’s 988 Lifeline crisis call centers would no longer be able to route calls to out-of-state back-up centers. California currently assesses an 8-cent fee on each telephone in this state, which supports the operations of the 988 service and associated call centers. To the extent that this bill increases workload and cost pressures for call centers beyond what can be supported by the surcharge, then additional General Fund support or an increase in the surcharge may be needed to comply with the mandates of this bill and meet the aims of the 988 service.

-- END --

SENATE COMMITTEE ON LOCAL GOVERNMENT

Senator María Elena Durazo, Chair

2023 - 2024 Regular

Public benefits contracts: phone operator jobs

Precludes certain state and local agencies, along with 211 and 988 services, from contracting with out-of-state call centers and restricts the use of artificial intelligence and automated decision systems to replace a worker’s core job function.

Background

California-only call centers. When it comes to contracting with call centers that serve people enrolled in five specific public benefit programs, state law requires those call centers to be staffed solely by workers employed in California. This requirement, created by AB 2508 (Bonilla, 2012), applies to:

• California Work Opportunity and Responsibility to Kids (CalWORKs);
• CalFresh;
• Medi-Cal;
• Healthy Families; and
• California Healthcare Eligibility, Enrollment, and Retention System.

The call center restrictions do not apply to:

• Contracts or subcontracts that would violate the specific terms of the Agreement on Government Procurement of the World Trade Organization or any other bilateral or regional free trade agreement to which the State of California has consented;
• Contracts or subcontracts, including extensions, that relate to a request for proposal or bid that occurred before this bill takes effect, should it be signed into law;
• Places where it may conflict with federal law; and
• Contracts for a public benefit program between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care (DMHC) and any subcontract performed under that contract, or a disability insurer or specialized health insurer regulated by the Department of Insurance (DOI), and any subcontract performed under that contract.

211 service. 211 is the nationally-designated phone number for community services information and referrals, often referred to as the 911 of non-emergency social, health, and human service needs. The national standard is for 211service to provide 24/7 information, referrals, and assistance, along with disaster information and resources. Here in California, 211 service is not available statewide and the service receives no direct state and federal funding for its operations or infrastructure. Rather, the service is operated by nonprofit groups, the majority of which are associated with the United Way and a few county governments, and must be approved by the California Public Utilities Commission (PUC). Right now, 42 of California’s 58 counties have 211 services, though service levels, use rates, and integration with health, housing, and disaster response systems vary by county. As for the remaining 16 counties:

• 1 is soon expected to provide 24/7 211 service (Tuolumne);
• 2 expect to file applications with the PUC soon (Colusa and Plumas);
• 2 have applications pending to provide 211 service only during disasters (Calaveras and Yuba);
• 5 do not have any services (Alpine, Amador, El Dorado, Modoc, and Sierra); and
• 6 have 211 service enabled but they do not currently provide services because legislatively-approved emergency funding expired (Del Norte, Lassen, Madera, Mono, Sutter, and Trinity).

Some of these centers use artificial intelligence systems to help workers assist callers, as well as to provide the public with “self-help” options. Additionally, during a disaster, 211 centers will sometimes contract with out-of-state call centers to provide assistance.

There are no general 211 service contracts or other agreements with the state, though some 211 providers do have contracts with different state agencies for special services that in some way use the 211 hotline. For example, most or all of the centers participate in the Tobacco Cessation Program, where operators ask people calling 211 for other reasons if they want to enroll in the program. Other 211 centers have received Listos California grants that are given to communities that are both socially vulnerable and at high risk of being impacted by a fire, flood, earthquake, drought or heatwave.

988 service. 988 has been designated as the national three-digit dialing code that routes callers to the National Suicide Prevention & Mental Health Crisis Lifeline. The Lifeline provides confidential emotional support to people in suicidal crisis or emotional distress 24 hours a day, 7 days a week, across the United States. When people call the number, they get an automated message that features additional options before a person is routed to their local Lifeline network crisis center.

There are 12 regional Lifeline crisis call centers in California – many of which have served as local community crisis assistance centers since the 1960s. They became Lifeline centers in the early 2000’s when the National Suicide Prevention Lifeline began, then they became 988 centers when 988 became the national assistance system in 2022. When someone dials 988, which center they are directed to is based on the area code they are calling from. If a California center is unable to take the call or assist the caller, the caller is automatically routed to one of the state’s two back-up centers (in Sacramento and Los Angeles). If one of those centers can’t answer the call, it is routed to one of the designated national back-up centers (there are nearly 200 centers nationwide, but not all serve as back-up centers).

Between January 2023 and January 2024, California call centers averaged 29,030 calls per month and the in-state answer rate was 88% -- the highest in-state answer rate in the country – meaning 12% of the calls were routed to out-of-state 988 operators.

The 988 system also has a text feature and a chat feature. During the same 13-month timeframe, about 8,300 texts were sent each month and in-state response rate was 73%, while approximately 5,000 monthly chats were initiated ant the in-state response rate was 75%.

Eleven of the 12 California centers operate with a mix of volunteers and paid staff, though all centers have more paid employees than volunteers and one center employs an entirely paid staff. The size of the call center staff ranges from 7 to 145, with the average being 43. Nine of the 12 centers are run by non-profits, one in San Diego is privately operated, and centers in Santa Clara and Kern counties are operated by the county’s health & human services agency.

The National Suicide Hotline Designation Act (NSHD), which turned the 988 program into a national program, authorizes states to impose a fee on all telephone lines to help finance 988 services. After investing $20 million in the 988 infrastructure in 2021 – which the centers are still relying on – California authorized an 8-cent fee to be assessed on each telephone line in the state. This is expected to raise about $35 million a year, of which the 988 centers are expected to receive an estimated $13-$14 million. The remaining funds will be allocated to the California Office of Emergency Services to develop an interoperability system between the 911 and 988 systems and some funds will be allocated to the California Health & Human Services Agency (CHHSA).

The author wants to ensure that when Californians call to get help with public benefit programs, they can reach a person, not just a machine, and that a person will be making the decisions as to whether or not the caller is entitled to receive public benefits.

Proposed Law

Senate Bill 1220:

• Limits the use of “artificial intelligence” (AI) and automated decisions systems (ADS) by state and local agencies, as well as 211 and 988 centers; and
• Expands the prohibition on using out-of-state call centers and workers to answer calls related to public benefit programs.

Limitations on use of artificial intelligence & automated decisions systems. SB 1220 prohibits state agencies, 211 and 988 service providers, and local agencies that receive state funding from using or contracting for the use of AI or ADS that eliminate or automate the core job function of any worker.

It requires that, in cases where a state or local agency funded in any part by state funding to provide services related to public benefits or services does use AI or ADS that impact the core job function of a worker, the agency is required to:

• Provide access to a human call worker if a caller asks to speak to a live person;
• Develop an impact assessment report that includes:
  • A detailed description of the technology and all its intended uses and roles in providing call center services;
  • A description of the data used by the technology, including the specific categories of data that will be processed as input and any data used to train the model that the technology relies upon;
  • A description of the types of outputs produced by the technology; and
  • Potential impacts on current or future employees, including, but not limited to, job loss, hour reductions and material changes to core job duties.
• The employees and their bargaining unit must be notified 60 days before the impact assessment report is drafted and the agency must do the work with the “direct, meaningful, and sustained involvement” of the employees who could be affected by the use of the technology.

Call centers. The bill expands the “California-only” call center restriction in existing law to include:

• All state agencies authorized to provide “public benefit programs;”
• 211 services;
• 988 services; and
• Any local government agency funded in any part by state funding to provide “services related to public benefits or services.”

Definitions. The bill creates a number of definitions, including:

Artificial intelligence (AI) is defined to mean “an engineered or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.”

An automated decision system (ADS) is defined to mean “a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision making and materially impacts natural persons.”

“Core job function” is defined as the “ability of a call center worker to fully exercise judgement throughout the service delivery process,” but states it does not include:

• Phone trees that filter callers to appropriate departments or language audios.
• Screening tools to provide initial suggestions on eligibility.
• AI or ADS used by workers to assist callers with public benefits.

An “impact assessment report” is defined as “a report by a contractor on AI, ADS, or related technologies used in the provision of call center services and its impact on call center workers.”

This bill deletes one exemption to the prohibition on using out-of-state call centers that was granted in 2012, thus applying all of SB 1220’s provisions to contracts regulated by DMHC and specialized health insurers regulated by DOI.

Finally, the bill states it does not apply to contracts or subcontracts, including extensions, that are signed after the bill takes effect on January 1, 2025, as long as the contract, subcontract, or extension stems from a request for proposal or bid that occurred before January 1, 2025.

Comments

1. Purpose of the bill. According to the author, “The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving the most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not artificial intelligence.”

2. Public benefits vs. public benefit programs. Impact on local governments could be vast. How this bill would affect state agencies and state public benefit programs is easier to envision because the bill specifically defines “state agency” and “public benefit programs” for the purposes of how and when AI and ADS can be used. However, for local governments, the description is very open-ended and could have wide-ranging implications.

The definition of “public benefit programs” applies to all “public benefits” provided by a local government agency funded in any part by state funding, but “public benefits” isn’t defined. Arguably, everything a local government does provides some type of a “public benefit” – police and fire services, the ability to rent a local park for a birthday party, paying city service bills online or by phone, youth recreational league sign-ups, summertime educational programs, and much more.

SB 1220 prevents a local agency that receives any state funds and provides call center services related to public benefits or services from using AI or ADS if doing so would eliminate or automate the core job function of any employee, meaning that as written, the bill could place constraints on the vast majority of local government services.

Now, the bill does state “core job function” does not include:

• Phone trees that filter callers to appropriate departments or language audios.
• Screening tools to provide initial suggestions on eligibility.
• AI or ADS used by workers to assist callers with public benefits.

As a result, this bill would presumably have no impact on a state agency or local government that wants to replace an employee with, for example, a phone tree as noted above. However, such a limited list does mean that other uses of AI or ADS and other technologies not listed above cannot be used to replace an employee and if they are used in a way that impacts an employee, then an impact assessment report would have to be conducted.

To reduce the likelihood of unintended consequences to wide swaths of local government activities, the definition of public benefit programs could be narrowed to those public social service programs implemented by counties on behalf of the state that are defined in Welfare & Institutions Code, Division 9. This includes programs related to child care, Head Start, aid for the aged, blind, and disabled, California veterans, California’s Newer Education & Well Being Program, and others.

3. Core job function. The main rationale for this bill is that humans – not AI, ADS, or other technologies – should be making decisions in terms of whether a person is entitled to receive certain benefits. One of the goals is to avoid something like what happened in Michigan in 2015 when the state adopted an automated system for its unemployment insurance benefit program. The system relied on flawed assumptions that were programmed into the system’s algorithm in an effort to identify fraud. As a result, the system charged 40,000 unemployment recipients with fraud, hitting them with repayment and fines with up to 400% interest. Reports are that up to 93% of the AI’s fraud charges were wrong.

In an attempt to prevent something similar from happening in California, SB 1220 prevents AI or ADS from being used to replace a worker’s ability “to fully exercise judgment throughout the service delivery process.” This, however, does not preclude an agency from using phone trees to filter callers to appropriate departments or language audios, using AI or ADS to screen callers and provide initial suggestions on eligibility, or having workers use AI or ADS to help callers with public benefits.

Instead of trying to define “core job function,” SB 1220 could focus instead on laying out a list of functions and decisions that state agencies and local governments must have made by human beings and not by machines. For example, the bill already allows certain phone tree functions, screening tools, and AI uses by workers, so perhaps it would be equally helpful to lay out specifically what functions AI, ADS, and other technologies would not be permitted to perform.

4. Impact on 211 services. The 211 service in California receives limited state grant funding and is run almost exclusively by nonprofit groups. The 211 operators use AI services to help them provide information to callers and they also use AI to provide callers with a “self-help” feature, allowing them to do research on their own. However, the 211 service occasionally contracts out work during emergencies, finding – for example – that it was quicker and more efficient during the wildfire emergencies in 2020 to utilize emergency or 211-type operators in other states than to try and locate a series of temporary workers or volunteers. Many local governments and non-profits are already struggling to be able to support these services and SB 1220’s contracting out and restrictions on using out-of-state operators may exacerbate that challenge. Should SB 1220 instead only apply the bill’s AI and ADS-related restrictions – and not the contracting out and call center restrictions – to the 211 services?

5. Impact on 988 services. The 988 system presents a different issue. Unlike the 211 service, 988 does receive state funding – it received $20 million in 2021 and is expected to soon begin receiving approximately $13-$14 million a year from an 8-cent surcharge levied on each phone line used in the state. CHHSA contracts with the 988 centers, which in turn hires its own employees. The total number of employees and volunteers is just over 500, with an estimated 300 being full-time equivalent employees.

Most, if not all, of the 988 centers are a hybrid mix of employees and volunteers who work on-site and remotely – both in California and out of state. When California 988 centers hire out-of-state workers, they are paid what California’s in-state employees are paid. According to the president of 988 California, because this wage goes further in many other states than it does in California, the organization often times has a much easier time getting out-of-state residents to apply for and accept these important positions. By making it more difficult or impossible to use out-of-state operations, SB 1220 could reduce the number of employees available to the system, potentially increasing the time it takes for a 988 center to answer and respond to someone calling for help. Furthermore, the 988 operation also rolls a certain amount of the calls, texts, and chats it receives – about 12% of calls, 27% of the texts, and 25% of the chats over the past 13 months – to 988 operators employed by other states’ call centers. Should SB 1220 only apply the AI and ADS-related restrictions – and not the bill’s contracting out and call center restrictions – to the 988 system?

6. Process for obtaining an exemption should be updated. Current law only applies the out-of-state call center restrictions on five specific health & human services-related programs. These programs can obtain an exemption by asking CHHSA or the CHBE board to find that:

• A prior solicitation for services was conducted and the bids received were priced unreasonably high as a result of including the in-state call center restrictions;
• A prior solicitation was conducted and fewer than two (which SB 1220 changes to zero) bids were received as a result of including the in-state call center restrictions; these provisions; or
• The services are needed in cases of emergency to protect the public health, welfare or safety.

This bill expands current law’s call center restrictions – and adds AI, ADS, and core job function definitions and restrictions – to also include all local government agencies that provide benefits or services, and to 211 and 988 centers. That means to get an exemption from the law, these entities would need to CHHSA or the CHBE board to approve the exemption.

Given that as written, the bill goes far beyond the five state health-related programs covered by current law, it is worth considering if an additional – or alternate – entity would be needed or more appropriate to handle these types of appeals that could come from all types (not just heath-related) programs run by local governments and/or the 211 and 988 centers.

7. Speed dating, committee style. The Senate Rules Committee double-referred this measure to the Senate Governmental Organization Committee, where it passed on a 10-3 vote on April 9^th, and then to the Senate Local Government Committee.

Support and Opposition

(4/12/24)

Support: California Labor Federation, AFL-CIO (Co-Sponsor)
California State Council of Service Employees International Union (SEIU California) (Co-Sponsor)
Orange County Employees Association
TechEquity Collaborative

Opposition: None received.

-- END --

SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION

Senator Bill Dodd

Chair

2023 - 2024 Regular

Bill No: 1220 Hearing Date: 4/9/2024

NOTE: This analysis reflects amendments that the author will be taking in committee.

SUBJECT: Public benefits contracts: phone operator jobs

DIGEST: This bill prohibits a state agency authorized to provide or enter into contracts relating to public benefit programs, as defined, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using Artificial Intelligence (AI), or Automated Decision Systems (ADS) that eliminate or automate the core job functions of a worker. If those state or local agencies use AI or ADS that impacts the core functions of a worker, the bill requires state and local agencies to comply with certain requirements, including notification to employees and the development of an impact assessment report, as specified.

ANALYSIS:

Existing law:

1. Prohibits, with specified exemptions, a state agency authorized to enter into contracts relating to public benefit programs, as defined, from contracting for services provided by a call center that directly serves applicants for, recipients for, or enrollees in, those public benefit programs with a contractor or subcontractor unless that contractor or subcontractor certifies in its bid for the contract that the contract, and any subcontract performed will be performed solely with workers employed in California.
2. Defines “public benefit programs,” for the above provisions to mean California Work Opportunity and Responsibility to Kids (CalWORKS), CalFresh, Medi-Cal, Healthy Families, and the California Healthcare Eligibility, Enrollment, and Retention System.
3. Defines “call center” for the above provisions to mean a building, facility, or operation where customer or client services or assistance is provided by telephone, fax, email, text, or Web-based interaction.
4. Provides an exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
5. Authorizes the state to terminate a contract relating to services provided by a call center if the contractor or subcontractor performs services with workers not employed in California.

This bill:

1. Prohibits a state agency, or a local agency funded in any part by state funding to provide call center services related to public benefits or services, from using AI, or ADS that eliminate or automate the core job functions of a worker.
2. Provides that if a state, or local agency funded in any part by state funding to provide services related to public benefits or services, uses AI or ADS that impact the core job function of a worker, then that agency shall comply with all of the following requirements:
   1. Provide access to a human call center worker upon request of a public caller.
   2. Develop an impact assessment report on the AI or ADS to be used in the call center. That report shall included all of the following elements:
      1. A detailed description of the technology and all its intended uses and roles in providing call center services.
      2. A description of the data used by the technology, including the specific categories of data that will be processed as input and any data used to train the model that the technology relies upon.
      3. A description of the types of outputs produced by the technology.
      4. Potential impacts on current or future employees, including, but not limited to, job loss, hour reductions, and material changes in core job duties.
3. Requires a state agency, if a state agency is authorized to provide or enter into contracts relating to public benefit programs, as defined, 211 services, or 988 services, and any local government agency funded in any part by state funding to provide services related to public benefits or services, to only provide services through a call center that directly serves callers with services performed solely with and by workers employed in California, as specified.
4. Requires the agency to notify employees potentially affected by the workplace technology, and their collective bargaining representative, at least 60 days before drafting the impact assessment.
5. Requires the agency to conduct all major phases of the impact assessment report with the direct, meaningful, and sustained involvement of employees impacted and their collective bargaining representative, if applicable. Sixty days before the ratification of the contract, the agency shall provide the report to employees potentially affected by the workplace technology, or their collective bargaining representative.
6. Extends the right to terminate a contract to a local government agency in the event a contractor or subcontractor performs the contract without workers or workers not employed in California, as specified.
7. Deletes a current exception for contracts between a state agency and a health care service plan or a specialized health care service plan regulated by the Department of Managed Health Care and for contracts between a state agency and a disability insurer or specialized health insurer regulated by the Department of Insurance.
8. Provides that the provisions of this bill shall not apply for contracts or subcontracts, including extensions of contracts or subcontracts, that result from a request for proposal or bid that occurred before January 1, 2025.
9. Defines “core job function” to mean the ability of a call center worker to fully exercise judgement throughout the service delivery process. This does not include:
   1. Phone trees that filter callers to appropriate departments or language audios.
   2. Screening tools to provide initial suggestions on eligibility.
   3. AI or digital tech used by workers to assist callers with public benefits.
10. Defines “impact assessment report” to mean a report by a contractor on AI or ADS technologies used in the provisions of call center services and its impact on call center workers.
11. Adds Covered California, Employment Development Insurance and other public benefits provided by a local government agency funded in any part by state funding to the definition of “public benefits programs.”
12. Defines “artificial intelligence,” or “AI,” to mean an engineering or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.
13. Defines “automated decision systems,” or “ADS” to mean a computational process derived from machine learning, statistical modeling, data analytics or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons.
14. Defines “state agency” to mean a state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services as described in Article 6.3 (commencing with Section 53123.1 of Chapter 1 of part 1 of Division 2 of Title 5 of the Government Code.)
15. Defines “worker” to mean a person.

Background

Author’s Statement. According to the author’s office, “SB 1220 will prohibit the state and local governments from outsourcing public benefit call center jobs to firms or employers using AI or other technology that completely replaces workers. This bill defines a worker as a person. It does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI. The purpose of this bill is to ensure there will be human assistance available when people call to receive help with public benefit programs, serving most vulnerable Californians. This bill does not preclude AI from assisting workers with their job tasks, but this bill clarifies that a worker must be a human being and not AI.”

Artificial Intelligence. AI refers to the development of computer systems that can perform tasks typically requiring human intelligence. These tasks include learning, reasoning, problem-solving, perception, and natural language understanding. AI enables computers to learn from data and experiences, allowing them to handle complex tasks without explicit programming. AI has the potential to revolutionize numerous sectors, including healthcare, finance, and transportation, by improving efficiency, reducing costs, and enhancing decision-making capabilities.

One of the most advanced and well-known AI systems, GPT-4, was developed by OpenAI and released on Pi Day (3/14), 2023. GPT-4 has demonstrated remarkable capabilities compared to its predecessor, GPT-3.5, which had been released to the world just six months prior. For example, GPT-4 scored around the top 10% on a simulated bar exam, while GPT-3.5 scored around the bottom 10%.

GPT-4 also scored a five out of five on several Advanced Placement (AP) exams, including Art History, Biology, Environmental Science, Macro- and Microeconomics, and Psychology, showcasing a knowledge across a diverse range of subjects. GPT-4 is particularly impressive at reasoning, scoring in the 93^rd percentile on the SAT Evidence-Based Reading & Writing exam and in the 99^th percentile on the Graduate Record Examination (GRE) Verbal exam.

The remarkable advancements in AI, exemplified by GPT-4's impressive performance on various professional and academic benchmarks, showcase its potential for practical applications in diverse fields. From law and education to science and technology, AI's ability to understand complex tasks and generate nuanced responses could lead to transformative solutions and new insights across multiple disciplines.

There are different types of AI, such as rule-based systems to make decisions or carry out tasks, and machine learning systems, which learn from data and experience to improve their performance over time. Deep learning, a type of machine learning, utilizes neural networks that mimic the structure and function of the human brain. Neural networks are used in deep learning to process large amounts of data and learn to recognize patterns, make predictions, and perform other tasks with a high degree of accuracy.

AI is already being used in a wide range of applications, including virtual assistants, image and speech recognition, fraud detection, predictive analytics, and autonomous vehicles. AI excels at fraud detection and predictive analytics due to its ability to rapidly and accurately analyze large datasets, detect patterns and anomalies challenging for humans to discern, and learn from historical data to predict future events. By detecting potential fraudulent activity and predicting future outcomes, AI can help businesses and governments make more informed decisions, save time and resources, and prevent financial losses.

While AI has the potential to revolutionize government services and decision-making, it also presents several challenges and concerns. One major issue is the potential for bias in AI systems. AI algorithms are only as unbiased as the data they are trained on, and if that data is biased or incomplete, it can result in discriminatory outcomes. For example, if an AI system is used to assess job applicants, it may inadvertently discriminate against certain groups if the data used to train the system is biased towards certain demographics. This could perpetuate existing inequalities in the workplace. To minimize bias in AI systems, it is crucial to ensure that the data used to train these systems is representative of the target population.

Another concern is the potential for privacy violations. AI systems can collect and process vast amounts of personal data, and if not properly secured, that data could be vulnerable to misuse or hacking. This is particularly concerning in government contexts, where sensitive information such as criminal records, medical histories, and financial information is often collected and stored. Additionally, there is concern that AI could displace jobs in certain sectors, particularly those that rely heavily on repetitive or routine tasks. While AI has the potential to automate many tasks and increase efficiency, it could also lead to significant job losses and create new inequalities in the workforce.

Further, there is the risk of unintended consequences. AI systems are complex and difficult to fully understand, and it is possible that they could produce unexpected or unintended outcomes. This is particularly concerning in government contexts where decisions made by AI systems could have far-reaching consequences for individuals and communities. For example, if an AI system is used to determine eligibility for social services, errors or biases in the system could result in individuals being denied important benefits. As such, it is important that AI is developed and implemented in a responsible and ethical manner, with proper safeguards and oversight in place to mitigate these risks.

AI in Government Services. AI is already being used in various ways within California government, including in the areas of transportation, healthcare, public safety, natural resource management, and emergency preparation and response. For instance, in 2020, the University of California, San Diego (UCSD) developed an AI-powered tool that can analyze chest X-rays to detect potential cases of COVID-19. This tool helped healthcare professionals quickly identify patients who needed immediate care, leading to more efficient allocation of medical resources during the pandemic.

In 2019, the number of traffic fatalities in the United States was 36,096, which is more than 11.17 deaths per 100,000 population. These numbers underscore the critical need to improve traffic, pedestrian, and bicyclist safety. The AI Pedestrian Traffic Safety System – tested on 12 hours of traffic videos captured from existing traffic cameras in the city of Los Angeles – demonstrated more than 98% accuracy in counting pedestrians in real time. This project occurred as part of LA’s public commitment to the global initiative Vision Zero, which endeavors to reduce severe injuries and deaths in roadway collisions. Los Angeles’s specific Vision Zero goal is to eliminate all traffic deaths and serious injuries by 2025. At full scale, the system can cover all 56,000 miles of LA streets and in real time.

After the 2018 Camp Fire in California, AI technology was used to analyze satellite imagery and assess the extent of the damage caused by the fire. This information helped emergency responders prioritize their efforts, allocate resources more efficiently, and speed up the recovery process for affected communities.

However, it is important to acknowledge that there are potential risks and negative consequences associated with AI. Bias, privacy violations, job displacement, and unintended consequences are some of the concerns associated with AI. These issues must be addressed and mitigated to ensure that the benefits of AI are maximized while minimizing any potential risks.

In 2016, a study by ProPublica revealed that a widely used AI-based risk assessment tool, COMPAS, was biased against African-American defendants in predicting their likelihood of reoffending. The algorithm was found to be more likely to falsely flag black defendants as high-risk and white defendants as low-risk, perpetuating existing racial biases in the criminal justice system.

When Use of AI Goes Wrong. The author’s office points to a number of examples where AI or automation have replaced human workers and the results have been devastating. For example, in 2023, the National Eating Disorder (NEDA) replaced a human staff helpline with an AI chatbot named Tessa. The version of Tessa that was initially tested and studied was a rule-based chatbot, meaning it could only use a limited number of prewritten responses. The CEO of NEDA was quoted as saying, “we were cognizant of the fact that AI isn’t ready for this population and so all of the responses were pre-programmed.” Once Tessa started being used by NEDA it underwent “systems upgrade,” including an “enhanced questions and answer feature.” The feature uses generative AI, meaning it gives the chatbot the ability to use new data and create new responses. After the upgrade Tessa began to telling callers to avoid “unhealthy foods” and only eat “healthy” snacks, like fruit. Such advice can be very harmful to those with eating disorders. After Tessa was accused of giving harmful advice, NEDA was forced to suspend its use.

In another example, the State of Michigan adopted an automated system for their unemployment insurance benefit program, which relied on flawed assumptions that were programmed into the algorithm to identify fraud. As a result, the AI charged 40,000 unemployment recipients with fraud, charging them repayment and fines with up to 400% interest. The charges drove people into debt, destroyed credit, and lives. Reports state that up to 93% of the AI’s fraud charges were wrong, however much of the damage had already been done.

As Proposed to be Amended. As previously noted, this analysis reflects amendments that the author has committed to accepting in committee. A below is a brief explanation of the major amendments.

Amendment 1: Replace language related to employing sufficient workers as follows:

 Employ sufficient workers to respond to callers requesting human assistance. Provide access to a human call center worker upon request of a public caller.

Amendments 2: Update definition of “public benefit programs” as follows:

“Public benefit programs” means California Work Opportunity and Responsibility to Kids (CalWORKs), CalFresh, Medi-Cal, Healthy Families, the California Healthcare Eligibility, Enrollment, and Retention System, California Health Benefit Exchange, Employment Development Department insurance, and any other needs-based program or information and referral program that refers people to, or provides benefits funded in any part by, the State of California. Covered California, Employment Development Department Insurance and other public benefits provided by a local government agency funded in any part by state funding.

Amendment 3: Delete the “or any state agency that provides benefits or services to the public,” delete prevailing wage language, and delete language pertaining to employing sufficient workers in section 12140.

12140. (a) (1) Notwithstanding any other law, any state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services, as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code, or any state agency that provides benefits or services to the public, and any local government agency funded in any part by state funding to provide services related to public benefits or services, shall only provide services through, or contract for services provided by, a call center that directly serves callers with services performed solely with and by workers employed in California, who are paid at least the prevailing minimum wage in the jurisdiction in California in which they are employed. A call center, as specified in this paragraph, shall also employ sufficient workers to respond to callers requesting human assistance.

Amendment 4: Add definition of state agency to clarify that this is only for public benefit programs, 211 services, and 988 services as follows:

“state agency” means a state agency authorized to provide or enter into contracts relating to public benefit programs, 211 services, or 988 services as described in Article 6.3 (commencing with Section 53123.1) of Chapter 1 of Part 1 of Division 2 of Title 5 of the Government Code,

Amendment 5: Delete “essential job function” and replace with “core job functions” throughout the bill and define “core job function” as follows:

“Core job function” means ability of a call center worker to fully exercise judgement throughout the service delivery process. This does not include:

(A) Phone trees that filter callers to appropriate departments or language audios.

(B) Screening tools to provide initial suggestions on eligibility.

(C) AI or digital tech used by workers to assist callers with public benefits.

Amendment 6: Redefine “Artificial Intelligence” and “automated decision systems” as follows:

(1) “Artificial intelligence,” or “AI,” means a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. an engineered or machine-based system that for explicit or implicit objectives, infers, from the input it receives, how to generate outputs that can influence physical or virtual environments and that may operate with varying levels of autonomy.

(2) “Automated decision systems,” or “ADS,” means a computational process, including one derived from machine learning, statistics, or other data processing or artificial intelligence techniques, that makes, or assists in making, decisions impacting workers or the public. a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons.

Amendment 7: Delete “other digital technologies” language throughout the bill to ensure that the focus of the bill is on AI and ADS.

Prior/Related Legislation

SB 892 (Padilla, 2024) requires CDT to develop and adopt regulations to create an AI risk management standard, consistent with specified publications regarding AI risk management, and prohibits state agencies, after the regulations are adopted, from contracting for an ADS unless the contract includes a clause that provides a complete risk assessment, as specified. (Pending in the Senate Governmental Organization Committee)

SB 893 (Padilla, 2024) requires Govops, the Governor’s Office of Business and Economic Development, and CDT to collaborate to establish the California Artificial Intelligence Research Hub to serve as a centralized entity to facilitate collaboration between government agencies, academic institutions, and private sector partners to advance AI Research and development that seeks to harness the technology’s full potential for public benefit while safeguarding privacy, advancing security, and addressing risks and potential harms to society, as specific. (Pending in the Senate Governmental Organization Committee)

SB 896 (Dodd, 2024) codifies much of EO N-12-23, ensuring state agencies advance safeguards and consumer protections around AI including notification requirements and the evaluation of automated decision-making systems being utilized by a state agency or department for risk potential prior to adoption. (Pending in the Senate Governmental Organization Comitee)

AB 2286 (Aguiar-Curry, 2024) prohibits the operation of an autonomous vehicle with a gross vehicle weight of 10,001 pounds or more on public roads for test purposes, transporting goods, or transporting passengers without a human safety operator physically present in the autonomous vehicle at the time of operation. (Pending in the Assembly Transportation Committee)

AB 2370 (Cervantes, 2024) prohibits AI from being used to replace community college faculty for purposes of providing academic instruction to, and to regular interaction with, students in a course instruction, and would authorize AI to only be used as a peripheral tool to support faculty in carrying out those tasks for uses such as course development, assessment, and tutoring. (Pending in the Assembly Privacy and Consumer Protection Committee)

SB 313 (Dodd, 2023) would have established the Office of AI, within the California Department of Technology, with the power and authority necessary to guide the design, use, and deployment of automated systems by a state agency, as specified. (Held on the Senate Appropriations Committee Suspense File)

AB 316 (Aguiar-Curry, 2023) would have restricted an autonomous vehicle with a gross vehicle weight of 10,001 pounds or more from being operated on public roads for testing purposes, transportation goods, or transporting passengers without a human safety operator physically present in the autonomous vehicle at the time of operation. (Vetoed by Governor Newsom)

AB 2508 (Bonilla, Chapter 824, Statutes of 2012) prohibits a state agency authorized to contract for public benefits programs from contracting for services provided by a call center that directly serves applicants for, recipients of, or enrollees in those programs, and that the work will be performed solely by workers employed in California.

FISCAL EFFECT: Appropriation: Fiscal Com.: Local:

SUPPORT:

California Labor Federation (Co-source)

TechEquity Collaborative

OPPOSITION:

None received

ARGUMENTS IN SUPPORT: According to the California Labor Federation, “the growth of AI has posed many challenges to California’s workforce. Employers often deploy technology for cost-cutting and without any worker input. The result is not only the elimination of good jobs, but the decline in the quality of public services. With the high volume of callers in need of support, public benefit programs such a Medi-Cal, the 988 Suicide Prevention & Mental Health Crisis Lifeline, and unemployment insurance assistance requires a support staff in a California that understands the issues facing the state’s most vulnerable populations. While not prohibiting the use of AI, SB 1220 will require agencies to produce impact assessment reports when utilizing AI, or ADS to guarantee that workers are at the center of the decision-making process during AI deployment.”

According to TechEquity Collaborative, “ultimately, AI simply cannot provide the level of empathy and effectiveness provided by a person. Public benefit call center workers have been on the front lines providing services to everyday Californians for years. During times of needs, ranging from global pandemic to intense natural disasters, public benefit call center workers have stood with people in need of guidance. Without protections from AI, call center workers face the erosion and eventual elimination of their jobs.”

DUAL REFERRAL: Senate Governmental Organization Committee & Senate Local Government Committee