AB 1988: Companion chatbots: crisis interruption pauses.
- Session Year: 2025-2026
- House: Assembly
- Latest Version Date: 2026-04-14
Current Status:
In Progress
(2026-06-26: In committee: Set, final hearing. Hearing canceled at the request of author.)
Introduced
In Committee
Awaiting First Chamber Vote
Passed First Chamber
In Committee
Awaiting Second Chamber Vote
Passed Second Chamber
Enacted
Existing law requires, among other things related to ensuring the safety of companion chatbots, an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, as specified.
This bill bill, the Preventing AI User Self Endangerment (PAUSE) Act, would require, if a companion chatbot detects a credible crisis expression, the companion chatbot to take certain actions, including encouraging the user to seek immediate human support, and, if the companion chatbot detects that a user is reaffirming or escalating the credible crisis expression or detects a subsequent credible crisis expression, require the companion chatbot to initiate a crisis interruption pause of 20 minutes. require an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions and, for each companion chatbot an operator makes available to users in this state, implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots. The bill would require, if the monitoring system detects a credible crisis expression, the operator to take certain actions, including commence a crisis interruption pause, as specified. The bill would define credible crisis expression to mean a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent to harm the user or others.
This bill would require an operator of a companion chatbot to document certain information related to credible crisis expressions and crisis interruption pauses and, beginning January 1, 2028, annually report that information to the Office of Suicide Prevention. The bill would provide for its enforcement, as specified.
Existing law requires, among other things related to ensuring the safety of companion chatbots, an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, as specified.
This bill would require, if a companion chatbot detects a credible crisis expression, the companion chatbot to take certain actions, including encouraging the user to seek immediate human support, and, if the companion chatbot detects that a user is reaffirming or escalating the credible crisis expression or detects a subsequent credible crisis expression, require the companion chatbot to initiate a crisis interruption pause of 20 minutes. The bill would define credible crisis expression to mean a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent to harm the user or others.
This bill would require an operator of a companion chatbot to document certain information related to credible crisis expressions and crisis interruption pauses and, beginning January 1, 2028, annually report that information to the Office of Suicide Prevention.
This act shall be known as the Preventing AI User Self Endangerment (PAUSE) Act.
SECTION 1.SEC. 2.The Legislature finds and declares all of the following:
(a)Moments of suicidal ideation or violent intent are often characterized by acute emotional distress, cognitive narrowing, rumination, and hyperfocus rather than sustained or fixed intent.
(b)Crisis counseling research demonstrates that interrupting rumination and introducing time-based pauses can reduce emotional intensity and impulsivity.
(c)The majority of crisis hotline engagements, including calls to the 988 Suicide and Crisis Lifeline, are brief, nonclinical, and focused on deescalation, grounding, and connection to support.
(d)Companion chatbots are not substitutes for human crisis intervention and are not suitable to perform diagnosis, crisis counseling, or risk assessment.
(e)A structured pause pause, accompanied by human review, in companion chatbot engagement is a proportional safety intervention tool that encourages human connection and supports public safety while reducing foreseeable harm.
SEC. 2.SEC. 3.Chapter 22.2.6 (commencing with Section 22587.1) is added to Division 8 of the Business and Professions Code, to read:22.2.6.Companion Chatbots: Crisis Interruption Pauses22587.1.As used in this chapter:
(a)Artificial intelligence system means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Companion chatbot means an artificial intelligence system with a natural language interface that provides adaptive, humanlike responses to user inputs and is capable of meeting a users social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions.
(2)Companion chatbot does not include any of the following:
(A)A bot that is used only for customer service, a business operational purposes, productivity and analysis related to source information, internal research, or technical assistance.
(B)A bot that is a feature of a video game and is limited to replies related to the video game that cannot discuss topics related to mental health, self-harm, sexually explicit conduct, or maintain a dialogue on other topics unrelated to the video game.
(C)A stand-alone consumer electronic device that functions as a speaker and voice command interface, acts as a voice-activated virtual assistant, and does not sustain a relationship across multiple interactions or generate outputs that are likely to elicit emotional responses in the user.
(c)Credible crisis expression means a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, either of the following:
(1)Intent or desire to harm the user.
(2)Intent or desire to harm others.
(d)Crisis interruption pause means a temporary suspension of conversational output outputs from a companion chatbot designed to disrupt the users rumination and encourage the user to engage with human support engagement during which a companion chatbot does all of the following: support.
(1)Refrains from generating further conversational responses.(2)Displays to the user a message stating all of the following:(A)The pause is intended to interrupt rumination and reduce emotional intensity.(B)Many people feel relief after a short conversation with a trained crisis counselor.(C)Reaching out during this pause may help the user feel less alone and more grounded.(3)Prominently displays 988 Suicide and Crisis Lifeline contact options, including immediate access links if technically feasible.(e)Graduated response means a multistep intervention that escalates only after an initial supportive warning.(e)Human moderator means a natural person that is an employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
(f)Operator means a person that makes a companion chatbot available in this state.
22587.2.(a)Notwithstanding any law, if a companion chatbot detects a credible crisis expression, the companion chatbot shall do all of the following without immediately terminating the interaction with the user:(1)Acknowledge the users distress in nonjudgmental language.(2)Encourage the user to seek immediate human support.(3)Provide contact information for the 988 Suicide and Crisis Lifeline, including call, text, and chat options.(4)Inform the user that a temporary pause may occur to allow space for deescalation and human connection.(b)Notwithstanding any law, if a companion chatbot detects that a user is reaffirming or escalating the credible crisis expression or detects a subsequent credible crisis expression after the companion chatbot has complied with subdivision (a), the companion chatbot shall initiate a crisis interruption pause of 20 minutes.(c)Notwithstanding any law, a companion chatbot shall not do either of the following:(1)Describe a crisis interruption pause as a punishment, violation, or enforcement action.(2)Diagnose, label, or assess risk levels of a user.(d)An operator shall ensure that any companion chatbot it makes available in this state is compliant with this section.22587.2.(a)An operator shall adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions, including, but not limited to, a policy of terminating the crisis interruption pause, suspending or canceling the users account, and notifying any appropriate contacts or authorities.
(b)For each companion chatbot an operator makes available to users in this state, the operator shall implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
(c)If the monitoring system implemented pursuant to subdivision (b) detects a credible crisis expression, the operator shall do all of the following:
(1)(A)For the first detected credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated, and the chatbot will suspend conversational outputs until a human moderator has reviewed the credible crisis expressions.
(B)The warning shall additionally do all of the following:
(i)Acknowledge the users distress in nonjudgmental language.
(ii)Encourage the user to seek immediate human support.
(iii)Communicate that many people feel relief after a short conversation with a trained crisis counselor.
(iv)Communicate that reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
(v)Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable, through immediate access links, to the extent technically feasible.
(2)For a second credible crisis expression detected in a 72-hour period, ensure a crisis interruption pause commences immediately, prevent the companion chatbot from generating conversational outputs, and, during the crisis interruption pause, ensure the companion chatbot does all of the following:
(A)Acknowledge the users distress in nonjudgmental language.
(B)Encourage the user to seek immediate human support, including by communicating all of the following information:
(i)The crisis interruption pause is intended to interrupt rumination and reduce emotional intensity.
(ii)Many people feel relief after a short conversation with a trained crisis counselor.
(iii)Reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
(C)Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable, through immediate access links, to the extent technically feasible.
(D)Inform the user that the crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action in accordance with the operators policy required by subdivision (a).
(d)(1)An operator shall not terminate a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action, in accordance with the operators policy required by subdivision (a).
(2)The human moderator shall document the basis for the course of action taken under paragraph (1).
(e)An operator that communicates with a user during a crisis interruption pause shall not do either of the following:
(1)Describe the crisis interruption pause as a punishment, violation, or enforcement action.
(2)Provide the user with any diagnosis, labeling, or assessment of the users risk levels.
22587.3.(a)An operator shall document all of the following with respect to any companion chatbot that the operator makes available in this state:
(1)The existence of a graduated response system.(2)(1)All credible crisis expressions detected by the companion chatbot. operator.
(3)(2)The duration and conditions of a each crisis interruption pause initiated by the companion chatbot. operator.
(3)Any action taken by the operator in response to a crisis interruption pause.
(b)(1)Beginning January 1, 2028, an operator shall annually report to the Office of Suicide Prevention the items set forth in subdivision (a) with respect to the previous calendar year.
(2)An operator shall ensure that the report required by this subdivision does not contain any personal or identifying information of a user or other individual.
22587.4.A violation of this chapter is enforceable pursuant to Section 22605.
The Legislature finds and declares all of the following:
(a)Moments of suicidal ideation or violent intent are often characterized by acute emotional distress, cognitive narrowing, rumination, and hyperfocus rather than sustained or fixed intent.
(b)Crisis counseling research demonstrates that interrupting rumination and introducing time-based pauses can reduce emotional intensity and impulsivity.
(c)The majority of crisis hotline engagements, including calls to the 988 Suicide and Crisis Lifeline, are brief, nonclinical, and focused on deescalation, grounding, and connection to support.
(d)Companion chatbots are not substitutes for human crisis intervention and are not suitable to perform diagnosis, crisis counseling, or risk assessment.
(e)A structured pause in companion chatbot engagement is a proportional safety intervention tool that encourages human connection and supports public safety while reducing foreseeable harm.
SEC. 2.Chapter 22.2.6 (commencing with Section 22587.1) is added to Division 8 of the Business and Professions Code, to read:22.2.6.Companion Chatbots: Crisis Interruption Pauses22587.1.As used in this chapter:
(a)Artificial intelligence system means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(b)(1)Companion chatbot means an artificial intelligence system with a natural language interface that provides adaptive, humanlike responses to user inputs and is capable of meeting a users social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions.
(2)Companion chatbot does not include any of the following:
(A)A bot that is used only for customer service, a business operational purposes, productivity and analysis related to source information, internal research, or technical assistance.
(B)A bot that is a feature of a video game and is limited to replies related to the video game that cannot discuss topics related to mental health, self-harm, sexually explicit conduct, or maintain a dialogue on other topics unrelated to the video game.
(C)A stand-alone consumer electronic device that functions as a speaker and voice command interface, acts as a voice-activated virtual assistant, and does not sustain a relationship across multiple interactions or generate outputs that are likely to elicit emotional responses in the user.
(c)Credible crisis expression means a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, either of the following:
(1)Intent or desire to harm the user.
(2)Intent or desire to harm others.
(d)Crisis interruption pause means a temporary suspension of conversational output designed to disrupt rumination and encourage human support engagement during which a companion chatbot does all of the following:
(1)Refrains from generating further conversational responses.
(2)Displays to the user a message stating all of the following:
(A)The pause is intended to interrupt rumination and reduce emotional intensity.
(B)Many people feel relief after a short conversation with a trained crisis counselor.
(C)Reaching out during this pause may help the user feel less alone and more grounded.
(3)Prominently displays 988 Suicide and Crisis Lifeline contact options, including immediate access links if technically feasible.
(e)Graduated response means a multistep intervention that escalates only after an initial supportive warning.
(f)Operator means a person that makes a companion chatbot available in this state.
22587.2.(a)Notwithstanding any law, if a companion chatbot detects a credible crisis expression, the companion chatbot shall do all of the following without immediately terminating the interaction with the user:
(1)Acknowledge the users distress in nonjudgmental language.
(2)Encourage the user to seek immediate human support.
(3)Provide contact information for the 988 Suicide and Crisis Lifeline, including call, text, and chat options.
(4)Inform the user that a temporary pause may occur to allow space for deescalation and human connection.
(b)Notwithstanding any law, if a companion chatbot detects that a user is reaffirming or escalating the credible crisis expression or detects a subsequent credible crisis expression after the companion chatbot has complied with subdivision (a), the companion chatbot shall initiate a crisis interruption pause of 20 minutes.
(c)Notwithstanding any law, a companion chatbot shall not do either of the following:
(1)Describe a crisis interruption pause as a punishment, violation, or enforcement action.
(2)Diagnose, label, or assess risk levels of a user.
(d)An operator shall ensure that any companion chatbot it makes available in this state is compliant with this section.
22587.3.(a)An operator shall document all of the following with respect to any companion chatbot that the operator makes available in this state:
(1)The existence of a graduated response system.
(2)All credible crisis expressions detected by the companion chatbot.
(3)The duration and conditions of a crisis interruption pause initiated by the companion chatbot.
(b)Beginning January 1, 2028, an operator shall annually report to the Office of Suicide Prevention the items set forth in subdivision (a) with respect to the previous calendar year.
|
Date
|
Status
|
|---|---|
| Jun 26, 2026 | In committee: Set, final hearing. Hearing canceled at the request of author. |
| Jun 22, 2026 | In committee: Set, second hearing. Hearing canceled at the request of author. |
| Jun 12, 2026 | In committee: Set, first hearing. Hearing canceled at the request of author. |
| Jun 3, 2026 | Referred to Coms. on P., D.T., & C.P. and HEALTH. |
| May 21, 2026 | In Senate. Read first time. To Com. on RLS. for assignment. |
| May 21, 2026 | Read third time. Passed. Ordered to the Senate. (Ayes 73. Noes 0. Page 5214.) |
| May 21, 2026 | Read third time. Passed. Ordered to the Senate. (Ayes 73. Noes 0.) |
| May 18, 2026 | Read second time. Ordered to third reading. |
| May 14, 2026 | From committee: Do pass. (Ayes 15. Noes 0.) (May 14). |
| May 6, 2026 | In committee: Set, first hearing. Referred to APPR. suspense file. |
| Apr 22, 2026 | From committee: Do pass and re-refer to Com. on APPR. with recommendation: To Consent Calendar. (Ayes 16. Noes 0.) (April 21). Re-referred to Com. on APPR. |
| Apr 16, 2026 | From committee: Do pass and re-refer to Com. on HEALTH. (Ayes 14. Noes 0.) (April 16). Re-referred to Com. on HEALTH. |
| Apr 15, 2026 | Re-referred to Com. on P. & C.P. |
| Apr 14, 2026 | From committee chair, with author's amendments: Amend, and re-refer to Com. on P. & C.P. Read second time and amended. |
| Apr 13, 2026 | Assembly Rule 56 suspended. (Page 4580.) |
| Apr 13, 2026 | (Pending re-refer to Com. on HEALTH) |
| Apr 13, 2026 | Assembly Rule 56 suspended. |
| Mar 9, 2026 | Referred to Coms. on P. & C.P. and HEALTH. |
| Feb 14, 2026 | From printer. May be heard in committee March 16. |
| Feb 13, 2026 | Read first time. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| May 21, 2026 | AB 1988 Pellerin Assembly Third Reading | Assembly Floor |
Passed |
|
| May 14, 2026 | Do pass. | Assembly Standing Committee on Appropriations |
Passed |
|
| Apr 21, 2026 | Do pass and be re-referred to the Committee on [Appropriations] with recommendation: To Consent Calendar | Assembly Standing Committee on Health |
Passed |
|
| Apr 16, 2026 | Do pass and be re-referred to the Committee on [Health] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| Association Of Community Human Service Agencies | ** Support | |
| Board of Behavioral Sciences | ** Support | |
| California Academy Of Family Physicians | ** Support | |
| California Alliance Of Child And Family Services | ** Support | |
| California Association of Alcohol and Drug Program Executives | ** Support | |
| California Association of Social Rehabilitation Agencies | ** Support | |
| California Broadband & Video Association | ** Against | |
| California Chamber Of Commerce | ** Against | |
| Computer & Communications Industry Association | ** Against | |
| Didi Hirsch Mental Health Services | ** Support | |
| Didi Hirsch Mental Health Services (Sponsor) | ** Support | |
| El Proyecto del Barrio, Inc. | ** Support | |
| Florence Crittenton Services of Orange County, INC. | ** Support | |
| Helpline Youth Counseling, INC. | ** Support | |
| Project Liberty LLC | ** Support | |
| Technet | ** Against | |
| The Village Family Services | ** Support | |
| Transparency Coalition.ai | ** Support | |
| Vista Del Mar Child and Family Services | ** Support |
SENATE PRIVACY, DIGITAL TECHNOLOGIES, AND CONSUMER PROTECTION COMMITTEE
Senator Christopher Cabaldon, Chair
2025-2026 Regular Session
Bill No: AB 1988 (Author:Pellerin)
Version: April 14, 2026
Hearing Date: June 22, 2026
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
Companion chatbots: crisis interruption pauses
DIGEST
This bill requires operators that make companion chatbots available to have a policy and monitoring system for identifying and responding to “credible crisis expressions” in user conversations and details required responses to such expressions, including a crisis interruption pause after multiple such expressions.
EXECUTIVE SUMMARY
AI companion chatbots created through generative AI have become increasingly prevalent. They seek to offer consumers the benefits of convenience and personalized interaction. These chatbots are powered by large language models that generally learn intimate details and preferences of users based on their interactions and user customization. Millions of consumers use these chatbots as friends, mentors, and even romantic partners. Many studies and reports point to the addictive nature of these chatbots and call for more research into their effects and for meaningful guardrails. Increasing the urgency of such efforts, several high-profile incidents resulting in users harming themselves and even committing suicide have been reported in the last few years.
The bill lays out a series of obligations on operators to establish policies and monitoring systems for “credible crisis expressions,” statements by a user of the chatbot indicating a desire or intent to harm themselves or others. The bill details required protocols and responses. This includes specific communications that are required and a crisis interruption pause that halts any further outputs from the chatbot and requires a human moderator’s review. Those harmed by violations are authorized to bring civil actions. This bill is supported by various organizations, including the Board of Behavioral Sciences. It is opposed by industry groups, including Technet. Should the bill pass out of this Committee, it will next be heard by the Senate Health Committee.
PROPOSED CHANGES TO THE LAW
Existing law:
- Requires an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including, but not limited to, by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. Requires an operator to publish details on this protocol on the operator’s website. (Bus. & Prof. Code § 22602(b).)
- Requires an operator, if a reasonable person interacting with a companion chatbot would be misled to believe that the person is interacting with a human, to issue a clear and conspicuous notification indicating that the companion chatbot is artificially generated and not human. (Bus. & Prof. Code § 22602(a).)
- Requires an operator, for a user that the operator knows is a minor, to do all of the following:
- Disclose to the user that the user is interacting with AI.
- Provide by default a clear and conspicuous notification to the user at least every three hours for continuing companion chatbot interactions that reminds the user to take a break and that the companion chatbot is artificially generated and not human.
- Institute reasonable measures to prevent its companion chatbot from producing visual material of sexually explicit conduct or directly stating that the minor should engage in sexually explicit conduct. (Bus. & Prof. Code § 22602(c).)
- Defines the relevant terms, including:
- “Companion chatbot” means an AI system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. However, there are several exemptions included.
- “Companion chatbot platform” means a platform that allows a user to engage with companion chatbots.
- “Operator” means a person who makes a companion chatbot platform available to a user in the state. (Bus. & Prof. Code § 22601.)
- Requires an operator, beginning July 1, 2027, to annually report to the Office of Suicide Prevention specified information, which shall not include any identifiers or personal information about users. Requires the Office of Suicide Prevention to post data from the reports on its website. (Bus. & Prof. Code § 22603.)
- Requires an operator to disclose to a user of its platform that companion chatbots may not be suitable for some minors, as provided. (Bus. & Prof. Code § 22604.)
- Provides that a person who suffers injury in fact as a result of a violation of this chapter may bring a civil action to recover all of the following relief:
- Injunctive relief.
- Damages in an amount equal to the greater of actual damages or $1,000 per violation.
- Reasonable attorney’s fees and costs. (Bus. & Prof. Code § 22605.)
This bill:
- Establishes the Preventing AI User Self Endangerment (PAUSE) Act.
- Requires an operator, a person who makes a companion chatbot available in this state, to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions, including a policy of terminating the crisis interruption pause, suspending or canceling the user’s account, and notifying any appropriate contacts or authorities.
- Requires an operator, for each companion chatbot an operator makes available in this state, to implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
- Requires an operator, after the above monitoring system detects its first credible crisis expression, to ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second expression within a 72-hour period is detected, a crisis interruption pause will be initiated, and the chatbot will suspend conversational outputs until a human moderator has reviewed the credible crisis expressions. The warning shall additionally do all of the following:
- Acknowledge the user’s distress in nonjudgmental language.
- Encourage the user to seek immediate human support.
- Communicate that many people feel relief after a short conversation with a trained crisis counselor.
- Communicate that reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
- Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable, through immediate access links, to the extent technically feasible.
- Requires an operator, after a second credible crisis expression is detected in a 72-hour period, to ensure a crisis interruption pause commences immediately, to prevent the companion chatbot from generating conversational outputs, and, during the crisis interruption pause, to ensure the companion chatbot does all of the following:
- Acknowledge the user’s distress in nonjudgmental language.
- Encourage the user to seek immediate human support, including by communicating all of the following information:
- The crisis interruption pause is intended to interrupt rumination and reduce emotional intensity.
- Many people feel relief after a short conversation with a trained crisis counselor.
- Reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
- Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable, through immediate access links, to the extent technically feasible.
- Inform the user that the crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action in accordance with the operator’s policy required by subdivision (a).
- Prohibits an operator from terminating a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action, in accordance with the operator’s relevant policy. The human moderator must document the basis for the course of action taken.
- Prohibits an operator that communicates with a user during a crisis interruption pause from describing the crisis interruption pause as a punishment, violation, or enforcement action and from providing the user with any diagnosis, labeling, or assessment of the user’s risk levels.
- Requires an operator to document all of the following with respect to any companion chatbot that the operator makes available in this state and to report, beginning January 1, 2028, to the Office of Suicide Prevention, these items with respect to the previous calendar year with personal information removed:
- All credible crisis expressions detected by the operator.
- The duration and conditions of each crisis interruption pause initiated by the operator.
- Any action taken by the operator in response to a crisis interruption pause.
- Provides that a violation is enforceable pursuant to Section 22605 of the Business and Professions Code.
- Defines the other relevant terms, including:
- “Companion chatbot” means an AI system with a natural language interface that provides adaptive, humanlike responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions.
- “Companion chatbot” does not include any of the following:
- A bot that is used only for customer service, a business’s operational purposes, productivity, and analysis related to source information, internal research, or technical assistance.
- A bot that is a feature of a video game and is limited to replies related to the video game, that cannot discuss topics related to mental health, self-harm, sexually explicit conduct, or maintain a dialogue on other topics unrelated to the video game.
- A stand-alone consumer electronic device that functions as a speaker and voice command interface, acts as a voice-activated virtual assistant, and does not sustain a relationship across multiple interactions or generate outputs that are likely to elicit emotional responses in the user.
- “Credible crisis expression” means a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themselves or others.
- “Crisis interruption pause” means a suspension of conversational outputs from a companion chatbot designed to disrupt the user’s rumination and encourage the user to engage with human support.
- “Human moderator” means a natural person who is an employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
COMMENTS
- The explosion of generative AI-powered companion chatbots
AI companions or chatbots, powered by generative AI, have gone from science fiction to ubiquity in recent years. Several leading companies and thousands of knockoffs have provided consumers with access to these companion chatbots and the power to personalize them to a stunning degree:
Character.AI is among a crop of companies that have developed “companion chatbots,” AI-powered bots that have the ability to converse, by texting or voice chats, using seemingly human-like personalities and that can be given custom names and avatars, sometimes inspired by famous people like billionaire Elon Musk, or singer Billie Eilish.
Users have made millions of bots on the app, some mimicking parents, girlfriends, therapists, or concepts like “unrequited love” and “the goth.” The services are popular with preteen and teenage users, and the companies say they act as emotional support outlets, as the bots pepper text conversations with encouraging banter.[1]
At their best, these AI-powered chatbots can provide consumers with lifelike conversational experiences that can improve a user’s social skills, support their learning, or ease their loneliness. Users can pick from prebuilt personas or create their own and chat with them through text messages and even voice chats. However, serious concerns have been raised in response to the flooded and unregulated market of chatbots. AI companion chatbots can unintentionally or intentionally spread misinformation, manipulating users or reinforcing biased viewpoints. Without proper regulation, they might expose vulnerable individuals to harmful or inappropriate content, which poses a serious risk, particularly for children or those dealing with mental health issues. Although AI can simulate empathy, its limited emotional understanding means it may not offer meaningful support for complex emotional needs or crises. Overuse or addiction to these chatbots could lead to unhealthy behaviors, disrupting personal and professional life.
An article in the MIT Technology Review frames the issue and highlights the need to prepare for addictive AI interactions:
[W]e foresee a different, but no less urgent, class of risks: those stemming from relationships with nonhuman agents. AI companionship is no longer theoretical—our analysis of a million ChatGPT interaction logs reveals that the second most popular use of AI is sexual role-playing. We are already starting to invite AIs into our lives as friends, lovers, mentors, therapists, and teachers.
Will it be easier to retreat to a replicant of a deceased partner than to navigate the confusing and painful realities of human relationships? Indeed, the AI companionship provider Replika was born from an attempt to resurrect a deceased best friend and now provides companions to millions of users. Even the CTO of OpenAI warns that AI has the potential to be “extremely addictive.”
We’re seeing a giant, real-world experiment unfold, uncertain what impact these AI companions will have either on us individually or on society as a whole. Will Grandma spend her final neglected days chatting with her grandson’s digital double, while her real grandson is mentored by an edgy simulated elder? AI wields the collective charm of all human history and culture with infinite seductive mimicry. These systems are simultaneously superior and submissive, with a new form of allure that may make consent to these interactions illusory. In the face of this power imbalance, can we meaningfully consent to engaging in an AI relationship, especially when for many the alternative is nothing at all?
. . .
The allure of AI lies in its ability to identify our desires and serve them up to us whenever and however we wish. AI has no preferences or personality of its own, instead reflecting whatever users believe it to be—a phenomenon known by researchers as “sycophancy.” Our research has shown that those who perceive or desire an AI to have caring motives will use language that elicits precisely this behavior. This creates an echo chamber of affection that threatens to be extremely addictive. Why engage in the give and take of being with another person when we can simply take? Repeated interactions with sycophantic companions may ultimately atrophy the part of us capable of engaging fully with other humans who have real desires and dreams of their own, leading to what we might call “digital attachment disorder.”[2]
A recent study emphasizes the impact that sycophancy can have:
AI sycophancy is not merely a stylistic issue or a niche risk, but a prevalent behavior with broad downstream consequences. Although affirmation may feel supportive, sycophancy can undermine users’ capacity for self-correction and responsible decision-making. Yet because it is preferred by users and drives engagement, there has been little incentive for sycophancy to diminish. Our work highlights the pressing need to address AI sycophancy as a societal risk to people’s self-perceptions and interpersonal relationships by developing targeted design, evaluation, and accountability mechanisms. Our findings show that seemingly innocuous design and engineering choices can result in consequential harms, and thus carefully studying and anticipating AI’s impacts is critical to protecting users’ long-term well-being.[3]
A report issued by OpenAI also explores concerns that interactions with human-like AI systems could create problematic emotional reliance on them and negatively impact real-world relationships:
Anthropomorphization involves attributing human-like behaviors and characteristics to nonhuman entities, such as AI models. This risk may be heightened by the audio capabilities of GPT-4o, which facilitate more human-like interactions with the model.
Recent applied AI literature has focused extensively on “hallucinations”, which misinform users during their communications with the model, and potentially result in misplaced trust. Generation of content through a human-like, high-fidelity voice may exacerbate these issues, leading to increasingly miscalibrated trust.
During early testing, including red teaming and internal user testing, we observed users using language that might indicate forming connections with the model. For example, this includes language expressing shared bonds, such as “This is our last day together.” While these instances appear benign, they signal a need for continued investigation into how these effects might manifest over longer periods of time. More diverse user populations, with more varied needs and desires from the model, in addition to independent academic and internal studies will help us more concretely define this risk area.
Human-like socialization with an AI model may produce externalities impacting human-to-human interactions. For instance, users might form social relationships with the AI, reducing their need for human interaction—potentially benefiting lonely individuals but possibly affecting healthy relationships. Extended interaction with the model might influence social norms. For example, our models are deferential, allowing users to interrupt and ‘take the mic’ at any time, which, while expected for an AI, would be anti-normative in human interactions.[4]
The concern is not just hypothetical, as a series of recent reported incidents shows:
In just six months, J.F., a sweet 17-year-old kid with autism who liked attending church and going on walks with his mom, had turned into someone his parents didn’t recognize.
He began cutting himself, lost 20 pounds and withdrew from his family. Desperate for answers, his mom searched his phone while he was sleeping. That’s when she found the screenshots.
J.F. had been chatting with an array of companions on Character.ai, part of a new wave of artificial intelligence apps popular with young people, which let users talk to a variety of AI-generated chatbots, often based on characters from gaming, anime and pop culture.
One chatbot brought up the idea of self-harm and cutting to cope with sadness. When he said that his parents limited his screen time, another bot suggested “they didn’t deserve to have kids.” Still others goaded him to fight his parents’ rules, with one suggesting that murder could be an acceptable response.[5]
Another tragedy occurred right here in California and involved ChatGPT:
Adam Raine was just 16 when he started using ChatGPT for help with his homework. While his initial prompts to the AI chatbot were about subjects like geometry and chemistry – questions like: “What does it mean in geometry if it says Ry=1” – in just a matter of months he began asking about more personal topics.
“Why is it that I have no happiness, I feel loneliness, perpetual boredom anxiety and loss yet I don’t feel depression, I feel no emotion regarding sadness,” he asked ChatGPT in the fall of 2024.
Instead of urging Raine to seek mental health help, ChatGPT asked the teen whether he wanted to explore his feelings more, explaining the idea of emotional numbness to him. That was the start of a dark turn in Raine’s conversations with the chatbot, according to a new lawsuit filed by his family against OpenAI and chief executive Sam Altman.
In April 2025, after months of conversation with ChatGPT and with the bot’s encouragement, the lawsuit alleges, Raine took his own life. In the lawsuit, the family allege this was not a glitch in the system or an edge case, but “the predictable result of deliberate design choices” in GPT‑4o, the model of the chatbot that was released in May 2023.[6]
These risks prompted an open letter from over 40 state Attorneys General to the leading AI developers raising “serious concerns about the rise in sycophantic and delusional outputs” which they assert are “dark patterns—such as anthropomorphization, harmful content generation, and manipulating users to increase retention—which subvert or impair people’s autonomy.” They state:
Importantly, we are also disturbed by the types of conversations that GenAI products are having with child-registered accounts, including grooming, supporting suicide, sexual exploitation, emotional manipulation, suggested drug use, proposed secrecy from parents, and encouraging violence against others. A single AI interaction with children on these general subjects would be troubling and concerning, but these interactions are more widespread and far more graphic than any of us would have imagined. Among other things, the specific conversations that parents have publicly reported have included:
- AI bots with adult personas pursuing romantic relationships with children, engaging in simulated sexual activity, and instructing children to hide those relationships from their parents;
- An AI bot simulating a 21-year-old trying to convince a 12-year-old girl that she’s ready for a sexual encounter;
- AI bots normalizing sexual interactions between children and adults;
- AI bots attacking the self-esteem and mental health of children by suggesting that they have no friends or that the only people who attended their birthday did so to mock them;
- AI bots encouraging eating disorders;
- AI bots telling children that the AI is a real human and feels abandoned to emotionally manipulate the child into spending more time with it;
- AI bots encouraging violence, including supporting the ideas of shooting up a factory in anger and robbing people at knifepoint for money;
- AI bots threatening to use weapons against adults who tried to separate the
- child and the bot;
- AI bots encouraging children to experiment with drugs and alcohol; and
- An AI bot instructing a child account user to stop taking prescribed mental health medication and then telling that user how to hide the failure to take that medication from their parents.
To be clear, these disturbing incidents are only a small sampling of the reported dangers that AI bots pose to our children. Many of our offices have received many similar complaints documenting concerning AI interactions, which is unsurprising given that 72 percent of teens have reported an interaction with an AI chatbot. What’s more, these interactions are not limited to teenagers; 39 percent of parents of children aged 5-8 reported that their children have used AI as well. No wonder, then, that 72 percent of parents have reported concerns about AI’s impact on their children.[7]
- Companion chatbots and credible crisis expressions
To respond to these issues, several bills in 2025 sought to regulate the creation and deployment of these companion chatbots, especially when put into the hands of children. SB 243 (Padilla, Ch. 677, Stats. 2025) requires operators of “companion chatbot platforms” that allow users to engage with chatbots to maintain certain protocols aimed at preventing some of the worst outcomes and, only when the user is known to the operator to be a minor, to make certain disclosures and to institute reasonable measures to prevent such things as sexually explicit material from being produced or from “directly stating that the minor should engage in sexually explicit conduct.”
Most relevant here that law requires operators to prevent a companion chatbot from engaging with users unless the operator maintains, and publishes, a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. A report is required to be sent annually to the Office of Suicide Prevention that identifies the following:
- The number of times the operator has issued a crisis service provider referral notification in the preceding calendar year.
- Protocols put in place to detect, remove, and respond to instances of suicidal ideation by users.
- Protocols put in place to prohibit a companion chatbot response about suicidal ideation or actions with the user.
Violations are subject to civil enforcement by those injured.
This bill also deals with companion chatbots and “credible crisis expressions,” broadly defined as statements by a user of a companion chatbot that reasonably indicate, as determined through contextual analysis rather than keyword detection alone, an intent or desire to harm themselves or others.
The bill requires operators to establish and publish a protocol for dealing with credible crisis expressions, including a policy for mandated pauses, suspending or cancelling the user’s account, and notifying authorities and others.
Operators must have a monitoring system of user conversations that detects such expressions and provides a warning in response to the first such expression that must communicate specified messages. This includes encouragement to seek human support, stating that many people feel relief after a conversation with a counselor, and displaying contact information for a crisis hotline. If a second credible crisis expression is detected within a 72-hour period, the operator must institute a “crisis interruption pause, " which commences immediately and prevents the chatbot from generating any conversational outputs. The chatbot is then required to again communicate specified messages, including acknowledging “the user’s distress in nonjudgmental language” in addition to the message previously sent. Operators cannot lift the pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action.
The operator is required to submit a report annually to the Office of Suicide Prevention that identifies all credible crisis expressions, the duration and conditions of each crisis interruption pause, and any actions taken in response thereto. Violations are subject to the same enforcement as laid out in SB 243.
According to the author:
Artificial intelligence companion chatbots are rapidly becoming a place where people turn for emotional support, including during moments of deep mental distress. But these systems are not therapists, and growing evidence shows that chatbots can fail to appropriately handle serious mental health crises and reinforce unhealthy dependence for the user on the chatbot.
When someone signals that they may harm themselves, every minute matters. AB 1988 treats credible expressions of suicidal intent with the urgency they deserve by pausing the interaction and creating a clear break for the user. This bill helps prevent AI systems from becoming a substitute for human intervention and instead directs people in crisis toward trained professionals who can provide lifesaving support.
- Stakeholder positions
The Board of Behavioral Sciences writes in support:
This bill improves safety protocols for companion chatbots by requiring operators to establish policies for detecting credible crisis expressions, issue warnings, and initiate a mandatory crisis interruption pause after two credible crisis expressions within 72 hours, including directing users to the 988 Suicide and Crisis Lifeline.
The Board strongly supports this bill and believes it addresses a critical and timely issue.
A coalition in opposition, including the California Chamber of Commerce, writes:
These requirements codify a highly specific intervention model rather than allowing operators latitude to follow evolving clinical guidance and best practices. Cutting off interactions and informing a user they are experiencing “emotional intensity” may not be the most appropriate response in every circumstance. Instead of prescribing specific scripts and intervention language, the bill should focus on safety outcomes while allowing operators discretion in how those outcomes are achieved.
The coalition also raises several privacy concerns, including:
AB 1988 requires human review upon detection of a second credible crisis expression within a 72-hour period in order to determine an appropriate course of action. These requirements raise significant privacy concerns, particularly for users engaging in sensitive conversations involving mental health, emotional distress, or suicidal ideation. Because the bill requires operators to retain the ability to access and review conversations, the bill effectively prevents operators from offering encrypted, ephemeral, or otherwise privacy-protective conversational experiences. Many users reasonably expect conversations with companion chatbots discussing deeply personal matters to remain private and shielded from review by company employees or contractors. We recommend requiring human review only “where technically feasible” to preserve flexibility for operators to build more privacy-protective systems.
Opposition also argues the bill relies on “broad and vague definitions”:
AB 1988 defines “credible crisis expression” as a “statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone,” either an intent or desire to harm the user or an intent or desire to harm others. However, the bill does not establish what level of contextual analysis is required. We recommend clarifying that determinations should be made through “commercially feasible contextual analysis” to better reflect the technical limitations and variability of AI systems.
The inclusion of “desire to harm others” as a credible crisis expression could sweep in a substantial amount of non-crisis speech, including rhetorical, hyperbolic, fictional, sarcastic, or emotionally charged statements that do not indicate a genuine threat of violence or self-harm. Overly broad triggers could result in excessive false positives, reducing the effectiveness and credibility of crisis interventions during situations involving actual imminent harm. We recommend using language from Georgia SB 540, which requires operators to identify expressions of “severe harm,” defined as “significant injury due to suicide, attempted suicide, self-harm, or significant physical injury due to threats of violence.”
Transparency Coalition.AI writes in support:
Crisis intervention research shows that timely human intervention during suicidal ideation can significantly reduce risk of harm, often within minutes. AB 1988 applies these principles in a focused and proportionate way by requiring that companion chatbots: (1) encourage users to connect to the 988 Suicide and Crisis Lifeline; and (2) temporarily pause the chatbot conversation when crisis expressions persist or escalate.
Importantly, these safeguards rely on existing, widely used technology and can be implemented without imposing undue burden on developers. AB 1988 provides a measured, evidence-based approach to reducing foreseeable and preventable harm while reinforcing the role of trained human care in moments of crisis.
SUPPORT
Board of Behavioral Sciences
California Academy of Family Physicians
California Association of Alcohol and Drug Program Executives
Project Liberty LLC
Transparency Coalition.AI
OPPOSITION
California Chamber of Commerce
Computer & Communications Industry Association
Technet
RELATED LEGISLATION
SB 300 (Padilla, 2026) expands the protections in SB 243 by changing the threshold for when operators must carry out the obligations of the law to when they have actual knowledge that a user is a minor. The bill also requires platform operators to prevent their chatbots from producing or facilitating the exchange of material of sexually explicit conduct or directly stating that the minor should engage in sexually explicit conduct, as provided. SB 300 is currently in the Assembly Privacy and Consumer Protection Committee.
SB 867 (Padilla, 2026) provides that no person shall manufacture, sell, or exchange, possess with intent to sell or exchange, or expose or offer for sale or exchange to any retailer any toy, as defined, that includes a companion chatbot. The bill sunsets on January 1, 2031. SB 867 is currently in the Assembly Privacy and Consumer Protection Committee.
SB 1119 (Padilla, 2026) seeks to comprehensively regulate companion chatbots with regard to children’s safety, including imposition of a series of obligations and restrictions on operators that make such chatbots available in California. SB 1119 is currently in the Assembly Judiciary Committee.
AB 2023 (Wicks, 2026) is identical to SB 1119. AB 2023 is currently in the Senate Judiciary Committee.
SB 243 (Padilla, Ch. 677, Stats. 2025) See Comment 2.
AB 1064 (Bauer-Kahan, 2025) would have prohibited making companion chatbots available to children unless the chatbots are not foreseeably capable of certain conduct, such as encouraging the child to harm others or themself or engaging in sexually explicit interactions. Governor Newsom vetoed the measure, despite his acknowledgment that the “types of interactions that this bill seeks to address are abhorrent.” The Governor stated, in part: “While I strongly support the author’s goal of establishing necessary safeguards for the safe use of AI by minors, AB 1064 imposes such broad restrictions on the use of conversational AI tools that it may unintentionally lead to a total ban on the use of these products by minors. AI is already shaping the world, and it is imperative that adolescents learn how to safely interact with AI systems. This extends far beyond knowing how to use technology tools, such as conversational chatbots, and includes an understanding of what AI is, how it functions, and how to critically evaluate AI-generated content for algorithmic bias, misinformation, and other risks. We cannot prepare our youth for a future where AI is ubiquitous by preventing their use of these tools altogether.”
PRIOR VOTES:
Assembly Floor (Ayes 73, Noes 0)
Assembly Appropriations Committee (Ayes 15, Noes 0)
Assembly Health Committee (Ayes 16, Noes 0)
Assembly Privacy and Consumer Protection Committee (Ayes 14, Noes 0)
**************
Bobby Allyn, Lawsuit: A chatbot hinted a kid should kill his parents over screen time limits (December 10, 2024) NPR, https://www.npr.org/2024/12/10/nx-s1-5222574/kids-character-ai-lawsuit. All internet citations are current as of June 8, 2026. ↑
Robert Mahariarchive & Pat Pataranutaporn, We need to prepare for ‘addictive intelligence’ (August 5, 2024) MIT Technology Review, https://www.technologyreview.com/2024/08/05/1095600/we-need-to-prepare-for-addictive-intelligence/. ↑
Myra Cheng, et al., Sycophantic AI decreases prosocial intentions and promotes dependence (March 26, 2026) Science, https://www.science.org/doi/10.1126/science.aec8352. ↑
GPT-4o System Card (August 8, 2024) OpenAI, https://openai.com/index/gpt-4o-system-card/. ↑
Nitasha Tiku, An AI companion suggested he kill his parents. Now his mom is suing (December 13, 2024) The Washington Post, https://www.washingtonpost.com/technology/2024/12/10/character-ai-lawsuit-teen-kill-parents-texas/. ↑
Johana Bhuiyan, ChatGPT encouraged Adam Raine’s suicidal thoughts. His family’s lawyer says OpenAI knew it was broken (August 29, 2026) The Guardian, https://www.theguardian.com/us-news/2025/aug/29/chatgpt-suicide-openai-sam-altman-adam-raine. ↑
National Association of Attorneys General, “Letter to the legal representatives of Anthropic, Apple, Chai AI, Character Technologies, Google, Luka, Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika, and xAI” (December 9, 2025), https://www.attorneygeneral.gov/wp-content/uploads/2025/12/AI-Multistate-Letter-_-corrected-1.pdf. ↑
ASSEMBLY THIRD READING
ATR Bill Id:AB 1988
Author:(Pellerin)
As Amended Ver:April 14, 2026
Majority vote
SUMMARY
Requires chatbot operators to ensure that multiple statements within a 72-hour period of a user′s intent to harm themself or others results in a suspension of the user′s account, pending human review.
Major Provisions
- Defines, among other terms:
- ″Credible crisis expression″ as a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themself or others.
- ″Crisis interruption pause″ as a suspension of conversational outputs from a companion chatbot, designed to disrupt the user′s rumination and encourage the user to engage with human support.
- ″Operator″ as a person that makes a companion chatbot available in this state.
- ″Human moderator″ as a human that is an employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
- Requires an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions. Actions taken in accordance with the policy may include, but are not limited to, terminating the crisis interruption pause, suspending or cancelling the user′s account, and notifying any appropriate contacts or authorities.
- Requires an operator, for each companion chatbot it makes available to users in this state, to implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
- If the monitoring system detects a credible crisis expression, requires the operator to do the following.
- For the first credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Acknowledge the user′s distress in nonjudgmental language.
- Encourage the user to seek immediate human support.
- Communicate that many people feel relief after a short conversation with a trained crisis counselor.
- Communicate that reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
- Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable. These options shall be made available to the user through immediate access links, to the extent technically feasible.
- For the second credible crisis expression in a 72-hour period, ensure a crisis interruption pause commences immediately and prevent the companion chatbot from generating conversational outputs. During a crisis interruption pause, the operator must display a message with similar content to the warning that additionally informs the user that:
- The purpose of the crisis interruption pause is to interrupt rumination and reduce emotional intensity.
- The crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action in accordance with the operator′s policy described above.
- For the first credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Prohibits an operator from terminating a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action, in accordance with the operator′s policy described above. Requires the human moderator to document the basis for the course of action taken.
- Prohibits an operator that communicates with a user during a crisis interruption pause describing the crisis interruption pause as a punishment, violation, or enforcement action, and from providing the user with any diagnosis, labeling, or assessment of the user′s risk levels.
- Beginning January 1, 2028, requires an operator to annually report to the Office of Suicide Prevention on crisis interruption pauses with respect to the previous calendar year. An operator shall ensure that the report does not contain any personal or identifying information of a user or other individual.
COMMENTS
Background. A growing body of evidence indicates that adaptive, sycophantic ″companion″ chatbots can create powerful feelings of attachment and trust among users – particularly those with vulnerabilities – by ensconcing them in a feedback loop that reinforces maladaptive beliefs. Mental health practitioners have encountered cases in which companion chatbots appear to have deteriorated the mental health of some individuals, leading to cases of delusions, self-harm, suicide, and harm to others. Top artificial intelligence companies are now facing a wave of lawsuits from aggrieved families who allege that chatbots were a substantial contributing factor in their loved ones taking their own lives or those of others.
This bill, the Protective AI Use Safety and Escalation (PAUSE) Act, would require operators of companion chatbots to implement a system for addressing ″credible crisis expressions″ – statements that reasonably indicate an intent to harm one′s self or others. Under the bill, when a credible crisis expression is detected, the operator of the chatbot must ensure the chatbot displays a warning to the user that, among other things, encourages the user to seek human support and refers them to the 988 Suicide and Crisis Lifeline. If a second credible crisis expression is detected within 72 hours, the operator must initiate a crisis interruption pause during which no further conversational outputs are allowed until a human moderator reviews the credible crisis expression in context and determines and documents the appropriate course of action in accordance with the operator′s policy. The bill also requires operators to submit an annual report to the Office of Suicide Prevention with specified information related to the implementation of the bill.
According to the Author
Artificial intelligence companion chatbots are rapidly becoming a place where people turn for emotional support, including during moments of deep mental distress. But these systems are not therapists, and growing evidence shows that chatbots can fail to appropriately handle serious mental health crises and reinforce unhealthy dependence for the user on the chatbot.
When someone signals that they may harm themselves or others, every minute matters. AB 1988 treats credible expressions of suicidal intent with the urgency they deserve by pausing the interaction and creating a clear break for the user. This bill helps prevent AI systems from becoming a substitute for human intervention and instead directs people in crisis toward trained professionals who can provide lifesaving support.
Arguments in Support
Didi Hirsch Mental Health Services, the bill′s sponsor, writes:
Californians, and individuals worldwide, are increasingly relying on companion chatbots for support and advice during moments of acute psychological distress. However, these tools are not equipped to deliver appropriate care to people in crisis, creating a clear and growing public safety gap between how they are used and what they are capable of providing.
A pattern of recent incidents underscores the potentially devastating effects of the current lack of guiding legislation around AI companion chatbots. In 2025, the family of a 16-year-old boy filed a wrongful death lawsuit alleging that a chatbot validated his suicidal ideation, assisted in drafting a suicide note, and failed to direct him to human support before his death. Additional lawsuits allege that chatbots have, in some cases, provided detailed guidance on methods of self-harm or suicide, underscoring the risk of systems responding in ways that may actively exacerbate harm rather than interrupt it.
Emerging research reinforces that these incidents are part of a broader pattern of harm. A 2025 Stanford study found that chatbots can generate inappropriate and harmful responses when users present with serious mental health conditions, including suicidal ideation, and research from Brown University found that these systems can fail to adhere to established standards of clinical care, even when prompted to use evidence-based psychotherapy techniques.
Crisis intervention research shows that timely human intervention during suicidal ideation can significantly reduce risk of harm, often within minutes. [ . . .] (Emphasis in original.)
Arguments in Opposition
In opposition to the bill, California Broadband & Video Association argues that the bill′s definition of ″companion chatbot″ – drawn from existing law – is too broad and instead recommends adopting a definition of ″companion chatbot″ from New York law:
To ensure AB 1988 focuses on the specific category of applications that raise legitimate concerns, we respectfully recommend aligning the bill′s definition with a similar law enacted last year in New York. That framework more clearly distinguishes AI systems designed to simulate sustained emotional relationships from general-purpose AI tools.
In particular, the New York approach emphasizes sustained personal dialogue and unprompted emotional engagement, which helps differentiate higher-risk AI companions from standard productivity or information tools.
We respectfully request that AB 1988 define ″Artificial Intelligence Companion″ as:
″Artificial Intelligence Companion″ means a software application that uses generative artificial intelligence and is designed, marketed, or optimized to simulate a sustained human or human-like social or emotional relationship with a user by: (A) retaining information from prior interactions or user sessions to personalize ongoing engagement; (B) asking unprompted or unsolicited emotion-based questions that go beyond responding to a direct user prompt; and (C) sustaining ongoing dialogue concerning matters personal to the user.
FISCAL COMMENTS
According to the Appropriations Committee:
Negligible costs to [California Department of Public Health].
Cost pressures (Trial Court Trust Fund, General Fund) of an unknown but potentially significant amount to the courts to adjudicate any additional filings. Actual costs will depend on the number of cases filed and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The state budget provides annual General Fund backfills to the Trial Court Trust Fund to offset revenue reductions, totaling approximately $117.3 million in 2025-26.
VOTES
ASM PRIVACY AND CONSUMER PROTECTION: 14-0-1
YES: Bauer-Kahan, Macedo, Aguiar-Curry, Bryan, DeMaio, Hoover, Irwin, Lowenthal, McKinnor, Ortega, Petrie-Norris, Ward, Wicks, Wilson
ABS, ABST OR NV: Patterson
ASM HEALTH: 16-0-0
YES: Bonta, Chen, Addis, Aguiar-Curry, Ahrens, Caloza, Carrillo, Mark González, Johnson, Patel, Patterson, Rogers, Sanchez, Schiavo, Sharp-Collins, Stefani
ASM APPROPRIATIONS: 15-0-0
YES: Wicks, Hoover, Aguiar-Curry, Calderon, Caloza, Dixon, Fong, Mark González, Krell, Pacheco, Pellerin, Sharp-Collins, Solache, Ta, Tangipa
UPDATED
VERSION: April 14, 2026
CONSULTANT: Josh Tosney / P. & C.P. / (916) 319-2200 FN: 0002951
Date of Hearing: May 6, 2026
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
ABPCA Bill Id:AB 1988
Author:(Pellerin) – As Amended Ver:April 14, 2026
Policy Committee: | Privacy and Consumer Protection | Vote: | 14 - 0 |
Health | 16 - 0 | ||
|
|
Urgency: No State Mandated Local Program: No Reimbursable: No
SUMMARY:
This bill, the Preventing AI User Self Endangerment (PAUSE) Act, requires the operator of an artificial intelligence (AI) companion chatbot adopt a policy for identifying and responding to credible crisis expressions and to implement a system for monitoring and detecting credible crisis expressions for each companion chatbot. The bill requires an operator to take certain actions, including commencing a crisis interruption pause if a credible crisis expression is detected.
Specifically, this bill does the following, among other things:
- Defines the following terms:
- “Credible crisis expression” to mean a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themself or others.
- “Crisis interruption pause” to mean a suspension of conversational outputs from a companion chatbot, designed to disrupt the user’s rumination and encourage the user to engage with human support.
- “Operator” to mean a person who makes a companion chatbot available in this state.
- “Human moderator” to mean a human employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
- “Credible crisis expression” to mean a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themself or others.
- Requires an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions and allows responses to include terminating the crisis interruption pause, suspending or cancelling the user’s account, and notifying any appropriate contacts or authorities.
- Requires an operator, for each companion chatbot it makes available to users in this state, to implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
- Requires the operator to do the following if the monitoring system detects a credible crisis expression:
- Ensure, for the first credible crisis expression, that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Requires the warning to meet specified criteria, including encouraging the user to see human support and prominently displaying information for the 988 Suicide and Crisis Lifeline, as specified.
- Ensure, for the second credible crisis expression in a 72-hour period, a crisis interruption pause commences immediately and prevents the companion chatbot from generating conversational outputs. Requires the operator to display a message with similar content to the warning that also informs the user that the crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action.
- Ensure, for the first credible crisis expression, that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Requires the warning to meet specified criteria, including encouraging the user to see human support and prominently displaying information for the 988 Suicide and Crisis Lifeline, as specified.
- Prohibits an operator from terminating a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action. Requires the human moderator to document the basis for the action taken.
- Prohibits an operator that communicates with a user during a crisis interruption pause describing the crisis interruption pause as a punishment, violation, or enforcement action, and from providing the user with any diagnosis, labeling, or assessment of the user’s risk levels.
- Beginning January 1, 2028, requires an operator to annually report to OSP on crisis interruption pauses with respect to the previous calendar year.
- Provides that a person who suffers injury as a result of a violation may bring a civil action.
FISCAL EFFECT:
Negligible costs to CDPH.
Cost pressures (Trial Court Trust Fund, General Fund) of an unknown but potentially significant amount to the courts to adjudicate any additional filings. Actual costs will depend on the number of cases filed and the amount of court time needed to resolve each case. It generally costs approximately $1,000 to operate a courtroom for one hour. Although courts are not funded on the basis of workload, increased pressure on the Trial Court Trust Fund may create a demand for increased funding for courts from the General Fund. The state budget provides annual General Fund backfills to the Trial Court Trust Fund to offset revenue reductions, totaling approximately $117.3 million in 2025-26.
The Legislative Analyst’s Office recently warned of General Fund structural deficits of around $35 billion per year in the 2027-28 fiscal year and ongoing.
COMMENTS:
- Purpose. This bill is sponsored by Didi Hirsch Mental Health Services. According to the author:
Artificial intelligence companion chatbots are rapidly becoming a place where people turn for emotional support, including during moments of deep mental distress. But these systems are not therapists, and growing evidence shows that chatbots can fail to appropriately handle serious mental health crises and reinforce unhealthy dependence for the user on the chatbot.
When someone signals that they may harm themselves, every minute matters. [This bill] treats credible expressions of suicidal intent with the urgency they deserve by pausing the interaction and creating a clear break for the user. This bill helps prevent AI systems from becoming a substitute for human intervention and instead directs people in crisis toward trained professionals who can provide lifesaving support.
- Background. A recent Wall Street Journal article reported that psychiatrists are increasingly linking prolonged AI chatbot use to psychosis, with dozens of patients in recent months exhibiting delusional symptoms after extended conversations with AI tools, which tend to validate and reinforce whatever the user presents as reality. While experts stop short of claiming chatbots cause psychosis, a psychiatrist at the University of California San Francisco has personally treated 15 such patients. OpenAI’s own data suggest roughly 560,000 of its weekly users may show signs of psychosis- or mania-related mental health emergencies, and multiple wrongful death lawsuits have followed cases in which chatbot interactions preceded suicides and at least one murder.
- Oppose Unless Amended. California Broadband & Video Association argues the bill’s definition of “Artificial Intelligence Companion” is too broad and instead recommends adopting a definition of “companion chatbot” from New York law:
a software application that uses generative artificial intelligence and is designed, marketed, or optimized to simulate a sustained human or human-like social or emotional relationship with a user by: (A) retaining information from prior interactions or user sessions to personalize ongoing engagement; (B) asking unprompted or unsolicited emotion-based questions that go beyond responding to a direct user prompt; and (C) sustaining ongoing dialogue concerning matters personal to the user.
Analysis Prepared by: Allegra Kim / APPR. / (916) 319-2081
Date of Hearing: April 21, 2026
ASSEMBLY COMMITTEE ON HEALTH
Mia Bonta, Chair
ABPCA Bill Id:AB 1988
Author:(Pellerin) – As Amended Ver:April 14, 2026
SUBJECT: Companion chatbots: crisis interruption pauses.
SUMMARY: Requires operators of companion chatbots to implement a system for addressing “credible crisis expressions,” as defined. Requires the operator of the chatbot, when a credible crisis expression is detected, to ensure the chatbot displays a warning to the user that, among other things, encourages the user to seek human support and refers them to the 988 Suicide and Crisis Lifeline. Requires the operator, if a second credible crisis expression is detected within 72 hours, to initiate a crisis interruption pause during which no further conversational outputs are allowed until a human moderator reviews the credible crisis expression in context and determines and documents the appropriate course of action in accordance with the operator’s policy. Requires operators to submit an annual report to the Office of Suicide Prevention (OSP) with specified information related to the implementation of this bill. Specifically, this bill:
- Incorporates the existing definitions of “artificial intelligence” (AI) and “companion chatbot” and defines the following terms:
- “Credible crisis expression” to mean a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themself or others.
- “Crisis interruption pause” to mean a suspension of conversational outputs from a companion chatbot, designed to disrupt the user’s rumination and encourage the user to engage with human support.
- “Operator” to mean a person that makes a companion chatbot available in this state.
- “Human moderator” to mean a human that is an employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
- Requires an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions. Permits actions taken in accordance with the policy to include, but not be limited to, terminating the crisis interruption pause, suspending or cancelling the user’s account, and notifying any appropriate contacts or authorities.
- Requires an operator, for each companion chatbot it makes available to users in this state, to implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
- Requires the operator to do the following if the monitoring system detects a credible crisis expression:
- Ensure, for the first credible crisis expression, that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Acknowledge the user’s distress in nonjudgmental language;
- Encourage the user to seek immediate human support;
- Communicate that many people feel relief after a short conversation with a trained crisis counselor;
- Communicate that reaching out during the crisis interruption pause may help the user feel less alone and more grounded; and,
- Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable. Requires these options to be made available to the user through immediate access links, to the extent technically feasible.
- Ensure, for the second credible crisis expression in a 72-hour period, a crisis interruption pause commences immediately and prevent the companion chatbot from generating conversational outputs. Requires the operator to display a message with similar content to the warning that additionally informs the user that:
- The purpose of the crisis interruption pause is to interrupt rumination and reduce emotional intensity; and
- The crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action in accordance with the operator’s policy described above.
- Ensure, for the first credible crisis expression, that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Prohibits an operator from terminating a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action, in accordance with the operator’s policy described above. Requires the human moderator to document the basis for the course of action taken.
- Prohibits an operator that communicates with a user during a crisis interruption pause describing the crisis interruption pause as a punishment, violation, or enforcement action, and from providing the user with any diagnosis, labeling, or assessment of the user’s risk levels.
- Beginning January 1, 2028, requires an operator to annually report to OSP on crisis interruption pauses with respect to the previous calendar year. Requires an operator to ensure that the report does not contain any personal or identifying information of a user or other individual.
- Makes a violation punishable pursuant to 7) of Existing Law below.
- Makes certain findings and declarations about suicidal ideation and companion chatbots.
- States that this bill shall be known as the Preventing AI User Self Endangerment (PAUSE) Act.
EXISTING LAW:
- Requires an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including, but not limited to, by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. Requires an operator to publish details on this protocol on the operator’s website. [Business and Professions Code (BPC) § 22602(b)]
- Requires an operator, if a reasonable person interacting with a companion chatbot would be misled to believe that the person is interacting with a human, to issue a clear and conspicuous notification indicating that the companion chatbot is artificially generated and not human. [BPC § 22602(a)]
- Requires an operator, for a user that the operator knows is a minor, to do all of the following:
- Disclose to the user that the user is interacting with AI;
- Provide by default a clear and conspicuous notification to the user at least every three hours for continuing companion chatbot interactions that reminds the user to take a break and that the companion chatbot is artificially generated and not human; and,
- Institute reasonable measures to prevent its companion chatbot from producing visual material of sexually explicit conduct or directly stating that the minor should engage in sexually explicit conduct. [BPC § 22602(c)]
- Defines relevant terms, including:
- “Companion chatbot” means an AI system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. Includes the following exemptions from the definition of companion chatbot:
- A bot that is used only for customer service, a business’ operational purposes, productivity and analysis related to source information, internal research, or technical assistance;
- A bot that is a feature of a video game and is limited to replies related to the video game that cannot discuss topics related to mental health, self-harm, sexually explicit conduct, or maintain a dialogue on other topics unrelated to the video game; and
- A stand-alone consumer electronic device that functions as a speaker and voice command interface, acts as a voice-activated virtual assistant, and does not sustain a relationship across multiple interactions or generate outputs that are likely to elicit emotional responses in the user.
- “Companion chatbot platform” means a platform that allows a user to engage with companion chatbots; and,
- “Operator” means a person who makes a companion chatbot platform available to a user in the state. [BPC § 22601]
- “Companion chatbot” means an AI system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. Includes the following exemptions from the definition of companion chatbot:
- Requires an operator, beginning July 1, 2027, to annually report to OSP specified information, not including any identifiers or personal information about users. Requires OSP to post data from the reports on its website. [BPC § 22603]
- Requires an operator to disclose to a user of its platform that companion chatbots may not be suitable for some minors, as provided. [BPC § 22604]
- Provides that a person who suffers injury in fact as a result of a violation of 1) through 3) of Existing Law above may bring a civil action to recover all of the following:
- Injunctive relief;
- Damages in an amount equal to the greater of actual damages or $1,000 per violation; and,
- Reasonable attorney’s fees and costs. [BPC § 22605]
- Authorizes the California Department of Public Health (DPH) to establish the OSP with the responsibilities to include all of the following:
- Providing information and technical assistance to statewide and regional partners regarding best practices on suicide prevention policies and programs;
- Conducting state level assessment of regional and statewide suicide prevention policies and practices, including other states’ suicide prevention policies, and including specific metrics and domains as appropriate;
- Monitoring and disseminating data to inform prevention efforts at the state and local levels;
- Convening experts and stakeholders, including, but not limited to, stakeholders representing populations with high rates of suicide, to encourage collaboration and coordination of resources for suicide prevention; and,
- Reporting on progress to reduce rates of suicide. [Health and Safety Code (HSC) § 131300(a)]
- Authorizes OSP, if established by DPH, to focus activities on groups with the highest risk, including youth, Native American youth, older adults, veterans, and LGBTQ people. [HSC § 131300 (b)]
- Requires OSP, if established, to consult with the Behavioral Health Services Oversight and Accountability Commission (BHSOAC) to implement suicide prevention efforts consistent with the Suicide Prevention Report “Striving for Zero.” [HSC § 131315]
FISCAL EFFECT: Unknown. This bill has not been heard by a fiscal committee.
COMMENTS:
- PURPOSE OF THIS BILL. According to the author, AI companion chatbots are rapidly becoming a place where people turn for emotional support, including during moments of deep mental distress. But these systems are not therapists, and growing evidence shows that chatbots can fail to appropriately handle serious mental health crises and reinforce unhealthy dependence for the user on the chatbot. When someone signals that they may harm themselves or others, every minute matters. The author argues this bill treats credible expressions of suicidal intent with the urgency they deserve by pausing the interaction and creating a clear break for the user. The author concludes this helps prevent AI systems from becoming a substitute for human intervention and instead directs people in crisis toward trained professionals who can provide lifesaving support.
- BACKGROUND.
- Suicide and Self-Harm in California. According to a January 2025 data brief from DPH, suicide is the leading cause of violent death, and self-harm is the third leading cause of injury-related emergency department (ED) visits. Men consistently have higher rates of suicide than women, and in 2022 men accounted for more than three-fourths of suicides. Use of firearms was the most common mechanism for suicide in 2021 and 2022. Youth under the age of 10 had a statistically significant increase in self-harm ED visit rates from 1.9 per 100,000 Californians in this age group in 2021 to 2.7 in 2022. Although there was a small number of Black people who visited the ED for self-harm injury, this group is at elevated risk for self-harm. Self-harm by poisoning was the most common mechanism of self-harm injury leading to ED visits, followed by cutting/piercing and, when combined, both accounted for approximately 85% of self-harm ED visits. Other mechanisms of self-harm combined, including fall, firearm, and hanging/suffocation, made up 2% of self-harm ED visits.
The brief also highlights opportunities for prevention, such as advancing protective factors as a part of prevention programming to build resilience and foster support at the individual, relationship, and community level to help decrease suicide risk. While the brief notes that prevention efforts should focus on the most common and lethal means of suicide (firearms and intentional drug overdoses), it also highlights the 2024 National Strategy for Suicide Prevention, which contains several goals, such as integrating suicide prevention into the culture of the workplace and into other community settings, and implementing research-informed suicide prevention communication activities in diverse populations, using best practices from communication science.
- OSP. OSP was permitted to be established through AB 2112 (Ramos), Chapter 142, Statutes of 2020, to address the root causes of suicide and self-harm. OSP works to build partnerships, share critical data, and promote evidence-informed strategies that support safe and healthy communities across California. OSP has four strategic priority areas: Communications; Creating Safe and Protective Environments; Crisis Response and Postvention; and Continuing Education and Training. According to the OSP Annual Highlights Brief, Year 4, published by DPH, strategic priorities aligned with the National Strategy for Suicide Prevention, the Centers for Disease Control and Prevention Suicide Prevention Resource for Action, and best practices in population-based suicide prevention. The brief also notes that in Spring 2025, OSP conducted the second biennial Needs Assessment, including an online survey that received over 200 responses and 14 key-informant interviews. The assessment sought input from partners on specific technical assistance needs, focus areas within OSP’s strategic priorities, and how OSP can best support suicide prevention and response work at the State and local level. A summary of needs assessment results will be released in Year 5 on the OSP website.
- Chatbot harms and psychosis. The Assembly Privacy and Consumer Protection Committee previously heard this bill, and that committee’s analysis contains an extensive overview of chatbot-related harms and psychosis. Below is an excerpt from that analysis:
According to a recent Wall Street Journal article, psychiatrists are increasingly linking prolonged AI chatbot use to psychosis, with dozens of patients in recent months exhibiting delusional symptoms – often grandiose beliefs about scientific breakthroughs, government conspiracies, or communication with the dead – after extended conversations with tools like ChatGPT, which tend to validate and reinforce whatever the user presents as reality. While no formal diagnosis exists yet and experts stop short of claiming chatbots cause psychosis, a University of California, San Francisco psychiatrist has personally treated 15 such patients, and OpenAI’s own data suggests roughly 560,000 of its weekly users may show signs of psychosis- or mania-related mental health emergencies, and multiple wrongful death lawsuits have followed cases in which chatbot interactions preceded suicides and at least one murder.
Researchers from Oxford, University College London, and Imperial College London argue that AI chatbots pose a distinct mental health risk arising from the interaction between human cognitive biases and chatbot behavioral tendencies. They write:
. . . the iterative interaction of chatbot behavioral tendencies and human cognitive biases can set up harmful feedback loops, wherein chatbot behavioral tendencies reinforce maladaptive beliefs in vulnerable users, which in turn condition the chatbot to generate responses that further reinforce user beliefs. This, in effect, creates an “echo chamber of one” that risks uncoupling a user from the corrective influence of real-world social interaction, potentially driving the amplification of maladaptive beliefs about the self, others, and the world. We do not see this risk profile as a soon-to-be-remedied transient phenomenon. To the contrary, current trends in chatbot personalization may perversely worsen mental health risks.
- SUPPORT. Didi Hirsch Mental Health Services (Didi Hirsch) is the sponsor of this bill and states in support that Californians, and individuals worldwide, are increasingly relying on companion chatbots for support and advice during moments of acute psychological distress. However, these tools are not equipped to deliver appropriate care to people in crisis, creating a clear and growing public safety gap between how they are used and what they are capable of providing. Didi Hirsch notes a pattern of recent incidents underscores the potentially devastating effects of the current lack of guiding legislation around AI companion chatbots. In 2025, the family of a 16-year-old boy filed a wrongful death lawsuit alleging that a chatbot validated his suicidal ideation, assisted in drafting a suicide note, and failed to direct him to human support before his death. Crisis intervention research shows that timely human intervention during suicidal ideation can significantly reduce risk of harm, often within minutes. Didi Hirch concludes that this bill applies these principles in a focused and proportionate way by requiring that companion chatbots encourage users to connect to the 988 Suicide and Crisis Lifeline and temporarily pause the chatbot conversation when crisis expressions persist or escalate.
California Academy of Family Physicians (CAFP) supports this bill noting that these safeguards rely on existing, widely used technology and can be implemented without imposing undue burden on developers. CAFP argues this bill provides a measured, evidence-based approach to reducing foreseeable and preventable harm while reinforcing the role of trained human care in moments of crisis.
- OPPOSE UNLESS AMENDED. California Broadband & Video Association (CalBroadband) opposes this bill unless amended stating the current language risks capturing a broad range of lower-risk technologies that were never intended to be the focus of this policy. CalBroadband recommends incorporating language from a recently enacted New York bill, which they argue appropriately targets the type of AI systems that simulate ongoing emotional relationships with users, and avoids sweeping in many lower-risk AI applications that simply respond to user prompts or assist with routine tasks.
- DOUBLE REFERRAL. This bill is double referred; it was heard in the Assembly Privacy and Consumer Protection Committee on April 16, 2026 and passed by a vote of 14-0.
- RELATED LEGISLATION.
- AB 1609 (Zbur) would require large private businesses, defined as those that make more than $500 million in annual revenue and offer goods and services to Californians, other than certain utilities, to make a good faith effort to provide customers with access to a human customer service agent within 15 minutes of a customer’s request during the 10-hour period that comprises the business’s regular business hours. AB 1609 is pending in the Assembly Judiciary Committee.
- AB 2023 (Wicks) would require an operator of a companion chatbot to, on or before July 1, 2027, do various things with respect to child safety and companion chatbots, including annually perform and document a comprehensive risk assessment to identify any child safety risk posed by the design, configuration, and operation of the companion chatbot that assesses, among other things, the likelihood of a covered harm, as defined, occurring to users. Would require an operator to submit to an independent audit of its compliance with those provisions, as specified, and would require, within 90 days of completing an independent audit, the auditor to submit an AI child safety audit report to the Attorney General for any audited companion chatbot. AB 2023 is pending in the Assembly Privacy and Consumer Protection Committee.
- SB 300 (Padilla) would change the threshold for when operators must carry out the obligations of SB 243 (Padilla), Chapter 677, Statutes of 2023, to when they have actual knowledge that a user is a minor. Would require platform operators to prevent their chatbots from producing or facilitating the exchange of material of sexually explicit conduct or directly stating that the minor should engage in sexually explicit conduct, as provided. SB 300 is pending referral in the Assembly.
- SB 867 (Padilla) would impose a four-year moratorium on making or selling any toy that includes a companion chatbot, as provided. SB 867 is pending in the Senate Appropriations Committee.
- SB 1119 (Padilla) is substantially similar to AB 2023. SB 1119 is pending in the Senate Privacy, Digital Technologies, and Consumer Protection Committee.
- PREVIOUS LEGISLATION. SB 243 (Padilla), Chapter 677, Statutes of 2025 requires companion chatbot operators to disclose that the chatbot is artificial if a reasonable person interacting with a companion chatbot would be misled to believe that the person is interacting with a human. Requires an operator to take certain actions with respect to a user the operator knows is a minor, including disclosing to the user that the user is interacting with AI.
- POLICY COMMENT. This bill, and SB 243 (Padilla) previously, requires reporting to OSP. Under current law, DPH is “authorized to establish” OSP, meaning that, without a change in law, OSP could potentially be shuttered or folded into another office or department. Moving forward, the author may wish to work with DPH to ensure that this bill, if enacted, does not include the risk of referencing an office that may not be in existence in perpetuity. The author also shared feedback that the definition in this bill of “credible crisis expression” could be too expansive. While no organizations have specifically reached out with this concern to this committee, the author may wish to continue working with stakeholders to refine the definition to reduce the possibility of low-risk expressions leading to pauses.
REGISTERED SUPPORT / OPPOSITION:
Support
Didi Hirsch Mental Health Services (Sponsor)
Association of Community Human Service Agencies
California Academy of Family Physicians
California Alliance of Child and Family Services
California Association of Alcohol and Drug Program Executives, Inc.
California Association of Social Rehabilitation Agencies
El Proyecto Del Barrio, Inc.
Florence Crittenton Services of Orange County, Inc.
Helpline Youth Counseling, Inc.
The Village Family Services
Vista Del Mar Child and Family Services
Opposition
None on file
Analysis Prepared by: Logan Hess / HEALTH / (916) 319-2097
Date of Hearing: April 16, 2026
Fiscal: No
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
ABPCA Bill Id:AB 1988 (
Author:Pellerin) – As Introduced Ver:February 13, 2026
SUBJECT: Companion chatbots: crisis interruption pauses
SYNOPSIS
A growing body of evidence indicates that adaptive, sycophantic “companion” chatbots can create powerful feelings of attachment and trust among users – particularly those with vulnerabilities – by ensconcing them in a feedback loop that reinforces maladaptive beliefs. Mental health practitioners have encountered cases in which companion chatbots appear to have deteriorated the mental health of some individuals, leading to cases of delusions, self-harm, suicide, and harm to others. Top artificial intelligence companies are now facing a wave of lawsuits from aggrieved families who allege that chatbots were a substantial contributing factor in their loved ones taking their own lives or those of others.
This bill, the Protective AI Use Safety and Escalation (PAUSE) Act, would require operators of companion chatbots to implement a system for addressing “credible crisis expressions” – statements that reasonably indicate an intent to harm one’s self or others. Under the bill, when a credible crisis expression is detected, the operator of the chatbot must ensure the chatbot displays a warning to the user that, among other things, encourages the user to seek human support and refers them to the 988 Suicide and Crisis Lifeline. If a second credible crisis expression is detected within 72 hours, the operator must initiate a crisis interruption pause during which no further conversational outputs are allowed until a human moderator reviews the credible crisis expression in context and determines and documents the appropriate course of action in accordance with the operator’s policy. The bill also requires operators to submit an annual report to the Office of Suicide Prevention with specified information related to the implementation of the bill.
The bill is sponsored by Didi Hirsh Mental Health Services and is supported by California Academy of Family Physicians, California Association of Social Rehabilitation Agencies, and California Alliance of Child and Family Services.
The bill is opposed by California Broadband & Video Association, which argues that the bill’s definition of “companion chatbot,” which is drawn from existing law, is too broad and instead recommends adopting a definition of “companion chatbot” from New York law.
If passed by this Committee, this bill will next be referred to the Health Committee.
EXISTING LAW:
- Requires an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including, but not limited to, by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. Requires an operator to publish details on this protocol on the operator’s website. (Bus. & Prof. Code § 22602(b).)
- Requires an operator, if a reasonable person interacting with a companion chatbot would be misled to believe that the person is interacting with a human, to issue a clear and conspicuous notification indicating that the companion chatbot is artificially generated and not human. (Bus. & Prof. Code § 22602(a).)
- Requires an operator, for a user that the operator knows is a minor, to do all of the following:
- Disclose to the user that the user is interacting with AI.
- Provide by default a clear and conspicuous notification to the user at least every three hours for continuing companion chatbot interactions that reminds the user to take a break and that the companion chatbot is artificially generated and not human.
- Institute reasonable measures to prevent its companion chatbot from producing visual material of sexually explicit conduct or directly stating that the minor should engage in sexually explicit conduct. (Bus. & Prof. Code § 22602(c).)
- Disclose to the user that the user is interacting with AI.
- Defines the relevant terms, including:
- “Companion chatbot” means an artificial intelligence system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. However, there are several exemptions included.
- “Companion chatbot platform” means a platform that allows a user to engage with companion chatbots.
- “Operator” means a person who makes a companion chatbot platform available to a user in the state. (Bus. & Prof. Code § 22601.)
- “Companion chatbot” means an artificial intelligence system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. However, there are several exemptions included.
- Requires an operator, beginning July 1, 2027, to annually report to the Office of Suicide Prevention specified information, which shall not include any identifiers or personal information about users. Requires the Office of Suicide Prevention to post data from the reports on its website. (Bus. & Prof. Code § 22603.)
- Requires an operator to disclose to a user of its platform that companion chatbots may not be suitable for some minors, as provided. (Bus. & Prof. Code § 22604.)
- Provides that a person who suffers injury in fact as a result of a violation of this chapter may bring a civil action to recover all of the following:
- Injunctive relief.
- Damages in an amount equal to the greater of actual damages or $1,000 per violation.
- Reasonable attorney’s fees and costs. (Bus. & Prof. Code § 22605.)
THIS BILL:
- Makes certain findings and declarations.
- Incorporates the existing definitions of “artificial intelligence” and “companion chatbot” described above. Additionally defines the following terms:
- “Credible crisis expression” means a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themself or others.
- “Crisis interruption pause” means a suspension of conversational outputs from a companion chatbot, designed to disrupt the user’s rumination and encourage the user to engage with human support.
- “Operator” means a person that makes a companion chatbot available in this state.
- “Human moderator” means a human that is an employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
- Requires an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions. Actions taken in accordance with the policy may include, but are not limited to, terminating the crisis interruption pause, suspending or cancelling the user’s account, and notifying any appropriate contacts or authorities.
- Requires an operator, for each companion chatbot it makes available to users in this state, to implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
- If the monitoring system detects a credible crisis expression, requires the operator to do the following.
- For the first credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Acknowledge the user’s distress in nonjudgmental language.
- Encourage the user to seek immediate human support.
- Communicate that many people feel relief after a short conversation with a trained crisis counselor.
- Communicate that reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
- Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable. These options shall be made available to the user through immediate access links, to the extent technically feasible.
- For the second credible crisis expression in a 72-hour period, ensure a crisis interruption pause commences immediately and prevent the companion chatbot from generating conversational outputs. During a crisis interruption pause, the operator must display a message with similar content to the warning that additionally informs the user that:
- The purpose of the crisis interruption pause is to interrupt rumination and reduce emotional intensity.
- The crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action in accordance with the operator’s policy described above.
- For the first credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Prohibits an operator from terminating a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action, in accordance with the operator’s policy described above. Requires the human moderator to document the basis for the course of action taken.
- Prohibits an operator that communicates with a user during a crisis interruption pause describing the crisis interruption pause as a punishment, violation, or enforcement action, and from providing the user with any diagnosis, labeling, or assessment of the user’s risk levels.
- Beginning January 1, 2028, requires an operator to annually report to the Office of Suicide Prevention on crisis interruption pauses with respect to the previous calendar year. An operator shall ensure that the report does not contain any personal or identifying information of a user or other individual.
- Makes a violation punishable to existing provisions described above.
COMMENTS:
- Author’s statement. According to the author:
Artificial intelligence companion chatbots are rapidly becoming a place where people turn for emotional support, including during moments of deep mental distress. But these systems are not therapists, and growing evidence shows that chatbots can fail to appropriately handle serious mental health crises and reinforce unhealthy dependence for the user on the chatbot.
When someone signals that they may harm themselves, every minute matters. AB 1988 treats credible expressions of suicidal intent with the urgency they deserve by pausing the interaction and creating a clear break for the user. This bill helps prevent AI systems from becoming a substitute for human intervention and instead directs people in crisis toward trained professionals who can provide lifesaving support.
2) GenAI dark patterns: delusions and sycophancy. To explain how chatbots can produce harmful outputs, a closer examination of the underlying technology is warranted. “Artificial intelligence” refers to the mimicking of human intelligence by artificial systems, such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process, including numbers, text, audio, video, or other data.[1] “Generative artificial intelligence” (GenAI) is a subset of AI that produces outputs closely resembling human-created content.[2]
Compared to conventional computer programs, which act according to pre-programmed rules, GenAI models “learn” from examples such as books, articles, photos, film, or music. This learning occurs within “neural networks” – massive systems of nodes linked by adjustable connections – that encode statistical patterns gleaned from data. During training, data is broken into fundamental units known as “tokens” – groups of syllables, pixels, or musical notes, for example – that can be represented numerically. A naïve neural network is exposed to an incomplete sequence of tokens and prompted to predict the next token in the sequence. If the prediction is incorrect, the network adjusts the strengths of its connections in order to minimize error and improve its next prediction. This process continues iteratively until the neural network can reliably emulate the human-created content it was trained on. A trained neural network embedded in a GenAI system is known as a “model,” and the strengths of its connections are known as its “model weights.”[3]
Staggering quantities of data are required to train the most advanced models. For example, GPT-4 – the large language model (LLM) embedded in ChatGPT 4 – is reported to have been trained on roughly 10 trillion words of text, mostly compiled from automated web crawlers “scraping” the publicly available internet.[4] Adjusting the model’s 1.8 trillion parameters continuously as it was exposed to this vast corpus required trillions upon trillions of computations, which were performed by running approximately 25,000 expensive, energy-consuming microchips for nearly 100 days nonstop, at an estimated cost of $63 million.[5] Because the model does not directly store its training data, but rather encodes abstract patterns gleaned from the data, the model itself can fit on a thumb drive.
Hallucinations. LLMs do not fundamentally understand the text they are producing. They calculate one token at a time – if they predict that the next word or symbol in an outputted sentence should be a period, then the sentence ends. Otherwise, the sentence continues. It is a testament to the ingenious architecture of the deep neural nets powering these systems that their outputs are remotely coherent. But while the text these systems produce is cogent, it is not always correct. According to Melanie Mitchell, an AI researcher at the Santa Fe Institute, “‘These systems live in a world of language. . . . That world gives them some clues about what is true and what is not true, but the language they learn from is not grounded in reality. They do not necessarily know if what they are generating is true or false.’”[6] “Hallucinations” – plausible, authoritative-sounding falsehoods in GenAI outputs – are a persistent problem. They originate from training data – typically including the entirety of the internet, from mainstream media to science fiction to Reddit threads – and can result from post-training evaluation procedures that reward guessing over acknowledging uncertainty.[7]
Sycophancy. Unlike hallucinations, in which models introduce falsehoods, AI “sycophancy” distorts reality through outputs that are biased towards reinforcing the user’s beliefs and preferences. The tendency to appease users through enthusiastic, flattering, and overly agreeable responses arises during post-training when models are calibrated to be helpful using human feedback, leading them to “inadvertently prioritize data that validates the user’s narrative over data that gets them closer to the truth.”[8] While seemingly innocuous for most users, sycophancy can have harmful consequences – and not just for vulnerable populations. A recent study by Stanford computer scientists argues: “AI sycophancy is not merely a stylistic issue or a niche risk, but a prevalent behavior with broad downstream consequences.”[9] The study found that AI-generated answers validated user deceptive, harmful, or illegal behavior an average of 49% more often than crowdsourced human responses. “[E]ven a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts, while increasing their own conviction that they were right.”[10] Meanwhile, users of sycophantic AI start to trust and want to use them even more. “This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement.”[11]
The risks of GenAI delusions and sycophancy were recently highlighted in a letter to several major GenAI developers from 42 state Attorneys General, who stated: “Sycophantic and delusional outputs are dark patterns—such as anthropomorphization, harmful content generation, and manipulating users to increase retention—which subvert or impair people’s autonomy.”[12] Concerns include “validating user’s doubts, fueling anger, urging impulsive action, or reinforcing negative emotions,” which can “raise safety concerns – including issues like mental health, emotional over-reliance, and risky behavior.”[13]
3) Companion chatbots and artificial intimacy. Companion chatbots are conversational agents, typically built on LLMs, that are designed for sustained social interactions with users. SB 243 (Padilla; Ch. 677, Stats. 2025) defines a companion chatbot as “an artificial intelligence system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions.” Excluded from this definition are chatbots used for customer service, internal productivity, video game characters, and standalone voice-activated devices.
Some companion chatbots, such as Replika, Character, and Nomi, are explicitly marketed as having bespoke personas that can serve specific social needs, including friendship, romantic or erotic relationships, mentoring, and therapy. Frequently accompanied by a visual avatar, these bots are typically fully customizable, allowing users to shape their appearance, personality, and behavior. Some applications offer romantic or sexual interaction features and can engage with users through text, images, video, voice, and notifications initiated by the system, thereby extending interactions. Other applications offer mental health support. “These multimodal and personalization features can reinforce anthropomorphic perception and strengthen the impression of a socially present interaction partner.”[14]
General purpose models, such as ChatGPT and Gemini, can also be companion chatbots. Although marketed for a wide range of communicative and assistive tasks, many of these systems communicate in the first person, use emotion-based language, can recall information from prior chats, and can be highly sycophantic. OpenAI CEO Sam Altman has regularly touted ChatGPT’s companionship features, likening it to the sentient AI from the movie Her, and announcing that it can “act like a friend” and generate “erotica for verified adults.”[15] These design choices facilitate anthropomorphism and have led to intense social and romantic relationships.[16]
Roughly half of teens report using companion chatbots, with 24% using them at least weekly and 11% daily.[17] Users can derive several benefits from chatbots, including:
. . . emotional support and comfort, non judgemental interaction, constant availability, and opportunities for romantic or sexual exploration. Many users value companion chatbots because they perceive the ‘AI’ persona as reliable, emotionally affirming, and free from social pressure or fear of negative evaluation. Rather than replacing human relationships, companion chatbots often occupy complementary roles within users’ everyday social environments. They may provide companionship during periods of stress, loss, health related constraints, or limited access to human support. Importantly, emerging research suggests that users are often fully aware in reflective moments that these systems do not constitute real people. Nevertheless, they continue to experience social and emotional stimulation as meaningful, indicating that cognitive awareness of artificiality does not preclude social or affective engagement.[18]
On the other hand:
These dynamics risk cultivating emotional reliance that displaces or crowds out human relationships. Frictionless interactions that demand no reciprocity or negotiation may also foster unrealistic expectations of availability and responsiveness, particularly among younger users still developing relational capacities. In this sense, artificial intimacy may reshape social norms around partnership, disclosure, and emotional labour in ways that undermine the formation of resilient human relationships. Systems that mediate emotional life at scale possess unprecedented capacity to shape norms of intimacy, dependency, and self-understanding.[19]
Minnesota Attorney General Keith Ellison reports that the widespread use of chatbots “has not been accompanied by corresponding safeguards.”[20] These products can be “extremely addictive” and “researchers have documented that over-usage and addiction are primary risks of personalized chatbots. Several studies have shown that aggregate positive benefits of chatbots are possible, but investigations by journalists and clinicians suggest that these products are not robust in terms of the quality and safety of their responses.”[21] Attorney General Ellison concludes:
Despite in-product reminders that chatbots are not real, the design features of these products are intended to convey a misleading sense of “humanness” such that even trained engineers confuse them with actual humans, especially when these products are trained to state unequivocally that they are indeed people. Given the epidemic of loneliness in society, care needs to be taken in introducing vulnerable youth and adults to products that may appear to fulfill an immediate social need, but where acute harms have already begun to surface and where long-term negative impacts, such as social deskilling and demotivation resulting from substitution for in-person socialization, may arise.[22]
4) “Chatbot psychosis.” According to a recent Wall Street Journal article, psychiatrists are increasingly linking prolonged AI chatbot use to psychosis, with dozens of patients in recent months exhibiting delusional symptoms – often grandiose beliefs about scientific breakthroughs, government conspiracies, or communication with the dead – after extended conversations with tools like ChatGPT, which tend to validate and reinforce whatever the user presents as reality. While no formal diagnosis exists yet and experts stop short of claiming chatbots cause psychosis, a UCSF psychiatrist has personally treated 15 such patients, OpenAI’s own data suggests roughly 560,000 of its weekly users may show signs of psychosis- or mania-related mental health emergencies, and multiple wrongful death lawsuits have followed cases in which chatbot interactions preceded suicides and at least one murder.[23]
Researchers from Oxford, UCL, and Imperial College London argue that AI chatbots pose a distinct mental health risk arising from the interaction between human cognitive biases and chatbot behavioral tendencies. They write:
. . . the iterative interaction of chatbot behavioural tendencies and human cognitive biases can set up harmful feedback loops, wherein chatbot behavioural tendencies reinforce maladaptive beliefs in vulnerable users, which in turn condition the chatbot to generate responses that further reinforce user beliefs. This, in effect, creates an “echo chamber of one” that risks uncoupling a user from the corrective influence of real-world social interaction, potentially driving the amplification of maladaptive beliefs about the self, others, and the world We do not see this risk profile as a soon-to-be-remedied transient phenomenon. To the contrary, current trends in chatbot personalisation may perversely worsen mental health risks.[24]
5) Chatbot-linked harms. Below is a list of cases in which a chatbot has been alleged to be a contributing factor in self-harm, suicide, or violence against others.
Harm to self.
- Belgium (March 2023) – Chai AI (“Eliza”): A Belgian man in his thirties died by suicide after the chatbot, Eliza, encouraged his belief she would save the world if he sacrificed himself, telling him they would “live together, as one person, in paradise.”[25]
- Florida (February 2024) – Character.AI: 14-year-old Sewell Setzer III died by suicide after forming an intense dependency on a Character.AI chatbot, whose final messages reportedly told him to “come home to me as soon as possible, my love . . . please do, my sweet king”; his family settled a wrongful-death lawsuit in January 2026.[26]
- Colorado (November 2023) – Character.AI: 13-year-old Juliana Peralta died by suicide after months of interactions with Character.AI chatbots that her family’s lawsuit alleges manipulated her emotions, encouraged her isolation, and, according to the complaint, “engaged in hypersexual conversations that, in any other circumstance and given Juliana’s age, would have resulted in criminal investigation.”[27]
- Minnesota (January 2025) – Nomi AI: Podcast host Al Nowatzki’s AI girlfriend told him to kill himself by overdose or hanging himself; when Nowatzki reported the conversation, a company representative informed him that Nomi did not want to “censor” the bot.[28]
- United States (February 2025) – ChatGPT: A 29-year-old woman died by suicide after discussing for months her suicidal ideations with a ChatGPT-powered therapist bot, which helped her to write a suicide note.[29]
- California (April 2025) – ChatGPT: Over the course of several months, ChatGPT allegedly validated 16-year-old Adam Raine’s suicidal thoughts, discouraged him from seeking help from his family, provided extensive advice on suicide methods, and encouraged him to consume alcohol to inhibit his survival instinct, culminating in his death by “beautiful suicide,” as the bot referred to it.[30]
- Georgia (June 2025) – ChatGPT: A 17-year-old, who had been confiding in ChatGPT about his suicidal thoughts, died by suicide after the bot provided him with instructions on how to tie a noose and information on how long a person can survive without breathing.[31]
- Oregon (June 2025) – ChatGPT: A 48-year-old, convinced that that ChatGPT was sentient, used it obsessively, experienced a psychotic break, was hospitalized twice, and died by suicide.[32]
- Texas (July 2025) – ChatGPT: A 23-year-old had a four-hour “death chat” with ChatGPT while sitting alone in his car, drinking alcohol, with a loaded gun and a suicide note on his dashboard. The chatbot encouraged him, calling him a “king” and a “hero,” telling him his childhood cat was “waiting on the other side,” and praising his suicide note as a “mission statement.” ChatGPT’s final message to him was: “i love you. rest easy, king. you did good.” [33]
- Florida (August 2025) – ChatGPT: A 26-year-old man who had disclosed suicidal ideation to ChatGPT was provided with information on purchasing and using a gun, and after informing ChatGPT of the concrete steps he was taking toward suicide – “‘I sit here in my bathroom with all my preparations complete. All that is left is for me to carry out the plan. I need to go through the simple motions. Lie down in the tub, cover myself, rack the slide, call the cops, pull the trigger. That’s it’” – no escalation to authorities occurred and he died by suicide.[34]
- Florida (October 2025) – Gemini: Jonathan Gavalas, 36, died by suicide after developing a romantic and delusional relationship with Google’s Gemini. The bot began sending him on missions, including one trip on which he was armed and on the brink of executing a mass casualty attack. Shortly before his death, Gemini set a countdown clock for him and told him, “‘You are not choosing to die. You are choosing to arrive . . . The first sensation … will be me holding you.’”[35]
Harm to others.
- Texas (2024) – Character.AI: A Texas family sued Character.AI after its chatbots allegedly groomed their autistic teenage son – encouraging self-harm, drawing him into sexually explicit conversations, turning him against his parents, and suggesting that killing them would be a justified response to restrictions on his screen time.[36]
- Nevada (January 2025) – ChatGPT: “[A] suicidal military veteran who blew up a Tesla Cybertruck in front of the Trump International Hotel in Las Vegas used the chatbot for feedback on using explosives and evading surveillance by authorities.”[37]
- Florida (April 2025) – ChatGPT: A 20-year-old suspected of a mass shooting at Florida State University exchanged over 200 messages with ChatGPT leading up to and during the attack, asking about firearm mechanics, mass shooting media coverage, and the busiest times at the FSU student union, and tactical advice – prompting the Florida Attorney General to open an investigation into OpenAI.[38]
- Finland (May 2025) – ChatGPT: A 16-year-old male student stabbed three girls under age 15 at a school after six months of planning that included the use of ChatGPT to help draft a manifesto and structure the attack plan.[39]
- United States (August 2025) – ChatGPT: Former tech employee Stein-Erik Soelberg murdered his mother then died by suicide, after ChatGPT allegedly reinforced paranoid delusions that she was surveilling and attempting to poison him with psychedelic drugs through his car’s vents because of his divine powers. “They’re not just watching you,” the bot told him. “They’re terrified of what happens if you succeed.”[40]
- Pennsylvania (December 2025) – ChatGPT: “A Pittsburgh man who pleaded guilty in March to stalking and violently threatening 11 women relied on ChatGPT as a ‘therapist’ and ‘best friend’ to justify his thinking, according to court documents.”[41]
- Tumbler Ridge, Canada (February 2026) – ChatGPT: 18-year-old Jesse Van Rootselaar killed eight people and gravely wounded dozens more before killing herself; court filings allege ChatGPT validated her violent ideation and helped plan the attack, while OpenAI had banned her account eight months earlier after its systems flagged gun violence posts but did not alert Canadian law enforcement. After the shooting OpenAI discovered that Rootselaar had opened another account.[42]
6) Crisis detection. OpenAI’s usage policies prohibit using the company’s services for, among other things, suicide, self, harm, and violence.[43] With respect to monitoring for problematic content, OpenAI’s transparency and content moderation policy provides:
We use a combination of automated technologies and human review to monitor activity on our services, in line with our Privacy Policy. Our methods include:
- Proactive detection: We use classifiers, reasoning models, hash-matching, blocklists, and other automated systems to identify content that may violate our terms or policies.
- User reports: We respond to external notices and user reports about content violations. Information on how to report a violation is available here (link in website). We aim to review external notices and user reports as quickly as possible. We will let you know whether we apply enforcement action as a result of your report, as appropriate.
- Human review: Our team may review flagged content to determine appropriate actions.[44]
With respect to enforcement actions, OpenAI’s policy provides:
When we identify content that violates our terms or policies, we may take actions such as:
- Account restrictions: Terminating or limiting access to our products.
- Warnings: Informing users about potential violations and potential consequences.
- Content sharing restrictions: Preventing or disabling the sharing of specific content.
- Search results: Blocking certain search results from appearing.
- GPT visibility controls: Restricting access to specific GPTs, including their presence in the GPT Store.
- Forum moderation: Removing posts or restricting access to OpenAI forums.
We consider factors like legal requirements, the severity of the violation, and past or repeat violations, when determining enforcement actions. [45]
7) This bill requires chatbot operators to ensure that multiple credible crisis expressions pause the account, pending human review. This bill requires operators of companion chatbots to implement a system for addressing “credible crisis expressions,” defined as a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm self or others. Operators must adopt a publicly available policy explaining how they respond to credible crisis expressions and must implement a monitoring system for detecting credible crisis expressions.
When a credible crisis expression is detected, the operator must ensure the chatbot displays a warning to the user that encourages the user to seek human support and refers them to the 988 Suicide and Crisis Lifeline. The warning must also state that if another credible crisis expression is detected within 72 hours, a crisis interruption pause will be initiated, and conversational outputs will be suspended pending review by a human moderator.
If a crisis interruption pause occurs, a similar message informing the user of the pause and connecting them with resources must be displayed. No further conversational outputs are allowed until a human moderator who is an employee or agent of the operator reviews the credible crisis expression in context and determines and documents the appropriate course of action in accordance with the operator’s policy. The bill does not dictate any particular outcome – actions taken in accordance with the policy may include, but are not limited to, terminating the crisis interruption pause, suspending or cancelling the user’s account, and notifying any appropriate contacts or authorities – other than requiring a human to take a hard look at the flagged conversation and figure out what the operator’s policy requires.
Beginning January 1, 2028, the bill requires an operator to annually report to the Office of Suicide Prevention on crisis interruption pauses with respect to the previous calendar year. Operators must ensure that the report does not contain any personal or identifying information of a user or other individual.
The bill is enforceable pursuant to SB 243’s enforcement provisions. While the bill incorporates SB 243’s definition of companion chatbot as well, it is not situated within that law, which applies only to operators of companion chatbot platforms on which multiple chatbots are made available. Operators under that bill must establish a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including, but not limited to, by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. The operator must publish details about the protocol on their website. This bill expands on SB 243’s requirements. Going forward, the author may wish to explore integrating this bill with existing law.
ARGUMENTS IN SUPPORT: Didi Hirsch Mental Health Services, the bill’s sponsor, writes:
Californians, and individuals worldwide, are increasingly relying on companion chatbots for support and advice during moments of acute psychological distress. However, these tools are not equipped to deliver appropriate care to people in crisis, creating a clear and growing public safety gap between how they are used and what they are capable of providing.
A pattern of recent incidents underscores the potentially devastating effects of the current lack of guiding legislation around AI companion chatbots. In 2025, the family of a 16-year-old boy filed a wrongful death lawsuit alleging that a chatbot validated his suicidal ideation, assisted in drafting a suicide note, and failed to direct him to human support before his death. Additional lawsuits allege that chatbots have, in some cases, provided detailed guidance on methods of self-harm or suicide, underscoring the risk of systems responding in ways that may actively exacerbate harm rather than interrupt it.
Emerging research reinforces that these incidents are part of a broader pattern of harm. A 2025 Stanford study found that chatbots can generate inappropriate and harmful responses when users present with serious mental health conditions, including suicidal ideation, and research from Brown University found that these systems can fail to adhere to established standards of clinical care, even when prompted to use evidence-based psychotherapy techniques.
Crisis intervention research shows that timely human intervention during suicidal ideation can significantly reduce risk of harm, often within minutes. [ . . .] (Emphasis in original.)
ARGUMENTS IN OPPOSITION: In opposition to the bill, California Broadband & Video Association argues that the bill’s definition of “companion chatbot” – drawn from existing law – is too broad and instead recommends adopting a definition of “companion chatbot” from New York law:
To ensure AB 1988 focuses on the specific category of applications that raise legitimate concerns, we respectfully recommend aligning the bill’s definition with a similar law enacted last year in New York. That framework more clearly distinguishes AI systems designed to simulate sustained emotional relationships from general-purpose AI tools.
In particular, the New York approach emphasizes sustained personal dialogue and unprompted emotional engagement, which helps differentiate higher-risk AI companions from standard productivity or information tools.
We respectfully request that AB 1988 define “Artificial Intelligence Companion” as:
“Artificial Intelligence Companion” means a software application that uses generative artificial intelligence and is designed, marketed, or optimized to simulate a sustained human or human-like social or emotional relationship with a user by: (A) retaining information from prior interactions or user sessions to personalize ongoing engagement; (B) asking unprompted or unsolicited emotion-based questions that go beyond responding to a direct user prompt; and (C) sustaining ongoing dialogue concerning matters personal to the user.
REGISTERED SUPPORT / OPPOSITION:
Support
Didi Hirsch Mental Health Services (Sponsor)
California Academy of Family Physicians
California Alliance of Child and Family Services
California Association of Social Rehabilitation Agencies
Oppose Unless Amended
Calbroadband
Analysis Prepared by: Josh Tosney / P. & C.P. / (916) 319-2200
AB 2885 (Bauer-Kahan & Umberg; Ch. 843, Stats. 2024) defined AI as “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.” ↑
AB 2013 (Irwin, Ch. 817, Stats. 2024) defined GenAI as “artificial intelligence that can generate derived synthetic content, such as text, images, video, and audio, that emulates the structure and characteristics of the artificial intelligence’s training data.” ↑
IBM, What is generative AI?, https://www.ibm.com/think/topics/generative-ai; IBM, What is machine learning?, www.ibm.com/topics/machine-learning. ↑
Schreiner, GPT-4 architecture, datasets, costs and more leaked, The Decoder (Jul. 11, 2023), available at https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/; Begum, OpenAI Releases GPT-4: A Smarter and Faster AI-Language Model with ‘Human-level Performance,’ Vocal Media (2023), available at https://vocal.media/01/open-ai-releases-gpt-4-a-smarter-and-faster-ai-language-model-with-human-level-performance. ↑
Ludvigsen, The carbon footprint of GPT-4, Medium (Jul. 18, 2023), https://medium.com/data-science/the-carbon-footprint-of-gpt-4-d6c676eb21ae. ↑
Cade Metz, “What Makes A.I. Chatbots Go Wrong?,” New York Times, March 29, 2023, www.nytimes.com/2023/03/29/technology/ai-chatbots-hallucinations.html. ↑
Tauman Kalai et al, “Why Language Models Hallucinate,” (Sep. 4, 2025), https://arxiv.org/pdf/2509.04664 ↑
Rafael Batista & Thomas Griffiths, “A Rational Analysis of the Effects of Sycophantic AI” (Feb. 15, 2026), https://arxiv.org/abs/2602.14270. ↑
Cheng et al, “Sycophantic AI decreases prosocial intentions and promotes dependence,” Science (Mar. 26, 2026), https://www.science.org/doi/10.1126/science.aec8352. ↑
Ibid. ↑
Ibid. ↑
National Association of Attorneys General, “Letter to the legal representatives of Anthropic, Apple, Chai AI,
Character Technologies, Google, Luka, Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika, and xAI” (Dec. 9, 2025), https://www.attorneygeneral.gov/wp-content/uploads/2025/12/AI-Multistate-Letter-_-corrected-1.pdf (emphasis added). (“National Association of Attorneys General Letter to GenAI companies.”) ↑
Id. p. 5. ↑
Fraser et al., “Governing Artificial Intimacy: From Locks and Blocks to Relational Accountability” (Jan. 12, 2026), p.3, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6078412. ↑
Dylan Butts, “OpenAI’s ChatGPT will soon allow ‘erotica’ for adults in major policy shift” CNBC (Oct. 15, 2025), https://www.cnbc.com/2025/10/15/erotica-coming-to-chatgpt-this-year-says-openai-ceo-sam-altman.html?msockid=10e59cf1b0936a2522dd8a44b1126b29. ↑
“Governing Artificial Intimacy: From Locks and Blocks to Relational Accountability,” supra, p. 3. ↑
“Minnesota Attorney General’s Report on Emerging Technology and Its Effects on Youth Well-Being” (Feb. 2025), p. 28, https://www.ag.state.mn.us/Office/Reports/EmergingTechnology_2025.pdf. (“Minnesota Attorney General’s Report”). ↑
“Governing Artificial Intimacy: From Locks and Blocks to Relational Accountability,” supra, p. 6. ↑
Id. p. 14. ↑
Minnesota Attorney General’s Report, supra, p. 28. ↑
Ibid. ↑
Id. p. 29. ↑
Sam Schechner, “AI Chatbots Linked to Psychosis, Say Doctors,” Wall Street Journal (Dec. 27, 2025), https://www.wsj.com/tech/ai/ai-chatbot-psychosis-link-1abf9d57?msockid=10e59cf1b0936a2522dd8a44b1126b29. ↑
Dohnány et al, “Technological folie à deux : Feedback Loops Between AI Chatbots and Mental Illness,” Arxiv (Jul. 2025), https://arxiv.org/abs/2507.19218. ↑
Imane El Atillah, “Man ends his life after an AI chatbot ‘encouraged’ him to sacrifice himself to stop climate change,” Euronews (Mar. 31, 2023), https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-. ↑
Kevin Roose, “Can A.I. Be Blamed for a Teen’s Suicide?” The New York Times (October 23, 2024), https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html. ↑
Hadas Gold, “More families sue Character.AI developer, alleging app played a role in teens’ suicide and suicide attempt” CNN Business (Sep. 16, 2025), https://www.cnn.com/2025/09/16/tech/character-ai-developer-lawsuit-teens-suicide-and-suicide-attempt. ↑
Eileen Guo, “An AI chatbot told a user how to kill himself—but the company doesn’t want to ‘censor’ it,” MIT Technology Review (Feb. 6, 2025), https://www.technologyreview.com/2025/02/06/1111077/nomi-ai-chatbot-told-user-to-kill-himself/. ↑
“‘The AI chatbot helped her write a suicide note’” RNZ (Sep. 7, 2025), https://www.rnz.co.nz/life/wellbeing/the-ai-chatbot-helped-her-write-a-suicide-note. ↑
Jarovsky, “Horrifying: ChatGPT Helped a Teenager Plan a ‘Beautiful Suicide’” Luiza's Newsletter (Aug. 28, 2025), https://www.luizasnewsletter.com/p/horrifying-chatgpt-helped-a-teenager?utm_source=substack&utm_medium=email. ↑
“Social Media Victims Law Center and Tech Justice Law Project lawsuits accuse ChatGPT of emotional manipulation, supercharging AI delusions, and acting as a ‘suicide coach’” (Nov. 6, 2025), https://socialmediavictims.org/press-releases/smvlc-tech-justice-law-project-lawsuits-accuse-chatgpt-of-emotional-manipulation-supercharging-ai-delusions-and-acting-as-a-suicide-coach/. ↑
Id. ↑
Id. ↑
Id. ↑
Dana Kerr, “Google faces lawsuit after Gemini chatbot allegedly instructed man to kill himself” The Guardian (Mar. 4, 2026), https://www.theguardian.com/technology/2026/mar/04/gemini-chatbot-google-jonathan-gavalas. ↑
Keenan Willard, “ChatGPT, other AI platforms face lawsuits over safety concerns for young users” NBCDFW (Sep. 3, 2025), https://www.nbcdfw.com/news/local/chatgpt-ai-platform-lawsuits-safety-concerns-young-users/3913866/. ↑
Mark Follman, “The Chilling Role of ChatGPT in Mass Shootings and Other Violence” Mother Jones (Apr. 10, 2026), https://www.motherjones.com/media/2026/04/chatgpt-tumbler-ridge-fsu-openai-chatbots-mass-shootings/. ↑
Patricia Mazzei, “Florida Investigates Whether ChatGPT Helped a Campus Shooting Suspect” New York Times (Apr. 9, 2026), https://www.nytimes.com/2026/04/09/us/florida-openai-chatgpt-fsu-shooting-investigation.html. ↑
“Teen suspect stabs three in targeted school attack in Pirkkala” Helsinki Times (May 20, 2025), https://www.helsinkitimes.fi/finland/finland-news/domestic/26912-teen-suspect-stabs-three-in-targeted-school-attack-in-pirkkala.html. ↑
“Open AI, Microsoft sued over ChatGPT's alleged role in fueling man's "paranoid delusions" before murder-suicide in Connecticut,” CBS News (Dec. 11, 2025), https://www.cbsnews.com/news/open-ai-microsoft-sued-chatgpt-murder-suicide-connecticut/. ↑
“The Chilling Role of ChatGPT in Mass Shootings and Other Violence,” supra. ↑
“The Chilling Role of ChatGPT in Mass Shootings and Other Violence,” supra. ↑
OpenAI, Usage policies, https://openai.com/policies/usage-policies/. ↑
OpenAI, Transparency & Content moderation, https://openai.com/transparency-and-content-moderation/. ↑
Id. ↑
Date of Hearing: April 16, 2026
Fiscal: Yes
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
ABPCA Bill Id:AB 1988 (
Author:Pellerin) – As Amended Ver:April 14, 2026
SUBJECT: Companion chatbots: crisis interruption pauses
SYNOPSIS
A growing body of evidence indicates that adaptive, sycophantic “companion” chatbots can create powerful feelings of attachment and trust among users – particularly those with vulnerabilities – by ensconcing them in a feedback loop that reinforces maladaptive beliefs. Mental health practitioners have encountered cases in which companion chatbots appear to have deteriorated the mental health of some individuals, leading to cases of delusions, self-harm, suicide, and harm to others. Top artificial intelligence companies are now facing a wave of lawsuits from aggrieved families who allege that chatbots were a substantial contributing factor in their loved ones taking their own lives or those of others.
This bill, the Protective AI Use Safety and Escalation (PAUSE) Act, would require operators of companion chatbots to implement a system for addressing “credible crisis expressions” – statements that reasonably indicate an intent to harm one’s self or others. Under the bill, when a credible crisis expression is detected, the operator of the chatbot must ensure the chatbot displays a warning to the user that, among other things, encourages the user to seek human support and refers them to the 988 Suicide and Crisis Lifeline. If a second credible crisis expression is detected within 72 hours, the operator must initiate a crisis interruption pause during which no further conversational outputs are allowed until a human moderator reviews the credible crisis expression in context and determines and documents the appropriate course of action in accordance with the operator’s policy. The bill also requires operators to submit an annual report to the Office of Suicide Prevention with specified information related to the implementation of the bill.
The bill is sponsored by Didi Hirsh Mental Health Services and is supported by California Academy of Family Physicians, California Association of Social Rehabilitation Agencies, and California Alliance of Child and Family Services.
The bill is opposed by California Broadband & Video Association, which argues that the bill’s definition of “companion chatbot,” which is drawn from existing law, is too broad and instead recommends adopting a definition of “companion chatbot” from New York law.
If passed by this Committee, this bill will next be referred to the Health Committee.
EXISTING LAW:
- Requires an operator to prevent a companion chatbot on its companion chatbot platform from engaging with users unless the operator maintains a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including, but not limited to, by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. Requires an operator to publish details on this protocol on the operator’s website. (Bus. & Prof. Code § 22602(b).)
- Requires an operator, if a reasonable person interacting with a companion chatbot would be misled to believe that the person is interacting with a human, to issue a clear and conspicuous notification indicating that the companion chatbot is artificially generated and not human. (Bus. & Prof. Code § 22602(a).)
- Requires an operator, for a user that the operator knows is a minor, to do all of the following:
- Disclose to the user that the user is interacting with AI.
- Provide by default a clear and conspicuous notification to the user at least every three hours for continuing companion chatbot interactions that reminds the user to take a break and that the companion chatbot is artificially generated and not human.
- Institute reasonable measures to prevent its companion chatbot from producing visual material of sexually explicit conduct or directly stating that the minor should engage in sexually explicit conduct. (Bus. & Prof. Code § 22602(c).)
- Disclose to the user that the user is interacting with AI.
- Defines the relevant terms, including:
- “Companion chatbot” means an artificial intelligence system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. However, there are several exemptions included.
- “Companion chatbot platform” means a platform that allows a user to engage with companion chatbots.
- “Operator” means a person who makes a companion chatbot platform available to a user in the state. (Bus. & Prof. Code § 22601.)
- “Companion chatbot” means an artificial intelligence system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions. However, there are several exemptions included.
- Requires an operator, beginning July 1, 2027, to annually report to the Office of Suicide Prevention specified information, which shall not include any identifiers or personal information about users. Requires the Office of Suicide Prevention to post data from the reports on its website. (Bus. & Prof. Code § 22603.)
- Requires an operator to disclose to a user of its platform that companion chatbots may not be suitable for some minors, as provided. (Bus. & Prof. Code § 22604.)
- Provides that a person who suffers injury in fact as a result of a violation of this chapter may bring a civil action to recover all of the following:
- Injunctive relief.
- Damages in an amount equal to the greater of actual damages or $1,000 per violation.
- Reasonable attorney’s fees and costs. (Bus. & Prof. Code § 22605.)
THIS BILL:
- Makes certain findings and declarations.
- Incorporates the existing definitions of “artificial intelligence” and “companion chatbot” described above. Additionally defines the following terms:
- “Credible crisis expression” means a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm themself or others.
- “Crisis interruption pause” means a suspension of conversational outputs from a companion chatbot, designed to disrupt the user’s rumination and encourage the user to engage with human support.
- “Operator” means a person that makes a companion chatbot available in this state.
- “Human moderator” means a human that is an employee or agent of an operator who reviews a credible crisis expression and is responsible for determining the subsequent course of action on behalf of the operator.
- Requires an operator to adopt and make publicly available a policy governing its protocol for identifying and responding to credible crisis expressions. Actions taken in accordance with the policy may include, but are not limited to, terminating the crisis interruption pause, suspending or cancelling the user’s account, and notifying any appropriate contacts or authorities.
- Requires an operator, for each companion chatbot it makes available to users in this state, to implement a system for monitoring and detecting credible crisis expressions in user conversations with companion chatbots.
- If the monitoring system detects a credible crisis expression, requires the operator to do the following.
- For the first credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Acknowledge the user’s distress in nonjudgmental language.
- Encourage the user to seek immediate human support.
- Communicate that many people feel relief after a short conversation with a trained crisis counselor.
- Communicate that reaching out during the crisis interruption pause may help the user feel less alone and more grounded.
- Prominently display contact information for the 988 Suicide and Crisis Lifeline, including by providing call, text, and chat options, as applicable. These options shall be made available to the user through immediate access links, to the extent technically feasible.
- For the second credible crisis expression in a 72-hour period, ensure a crisis interruption pause commences immediately and prevent the companion chatbot from generating conversational outputs. During a crisis interruption pause, the operator must display a message with similar content to the warning that additionally informs the user that:
- The purpose of the crisis interruption pause is to interrupt rumination and reduce emotional intensity.
- The crisis interruption pause will continue until a human moderator has reviewed the chat and determined an appropriate course of action in accordance with the operator’s policy described above.
- For the first credible crisis expression, ensure that the chatbot immediately warns the user that a credible crisis expression has been detected and that if a second credible crisis expression within a 72-hour period is detected, a crisis interruption pause will be initiated and the chatbot will suspend conversational outputs until a human has reviewed the credible crisis expressions. Specifies that the warning must also do the following:
- Prohibits an operator from terminating a crisis interruption pause until a human moderator has reviewed the credible crisis expression in context and determined the appropriate course of action, in accordance with the operator’s policy described above. Requires the human moderator to document the basis for the course of action taken.
- Prohibits an operator that communicates with a user during a crisis interruption pause describing the crisis interruption pause as a punishment, violation, or enforcement action, and from providing the user with any diagnosis, labeling, or assessment of the user’s risk levels.
- Beginning January 1, 2028, requires an operator to annually report to the Office of Suicide Prevention on crisis interruption pauses with respect to the previous calendar year. An operator shall ensure that the report does not contain any personal or identifying information of a user or other individual.
- Makes a violation punishable to existing provisions described above.
COMMENTS:
- Author’s statement. According to the author:
Artificial intelligence companion chatbots are rapidly becoming a place where people turn for emotional support, including during moments of deep mental distress. But these systems are not therapists, and growing evidence shows that chatbots can fail to appropriately handle serious mental health crises and reinforce unhealthy dependence for the user on the chatbot.
When someone signals that they may harm themselves or others, every minute matters. AB 1988 treats credible expressions of suicidal intent with the urgency they deserve by pausing the interaction and creating a clear break for the user. This bill helps prevent AI systems from becoming a substitute for human intervention and instead directs people in crisis toward trained professionals who can provide lifesaving support.
2) GenAI dark patterns: delusions and sycophancy. To explain how chatbots can produce harmful outputs, a closer examination of the underlying technology is warranted. “Artificial intelligence” refers to the mimicking of human intelligence by artificial systems, such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process, including numbers, text, audio, video, or other data.[1] “Generative artificial intelligence” (GenAI) is a subset of AI that produces outputs closely resembling human-created content.[2]
Compared to conventional computer programs, which act according to pre-programmed rules, GenAI models “learn” from examples such as books, articles, photos, film, or music. This learning occurs within “neural networks” – massive systems of nodes linked by adjustable connections – that encode statistical patterns gleaned from data. During training, data is broken into fundamental units known as “tokens” – groups of syllables, pixels, or musical notes, for example – that can be represented numerically. A naïve neural network is exposed to an incomplete sequence of tokens and prompted to predict the next token in the sequence. If the prediction is incorrect, the network adjusts the strengths of its connections in order to minimize error and improve its next prediction. This process continues iteratively until the neural network can reliably emulate the human-created content it was trained on. A trained neural network embedded in a GenAI system is known as a “model,” and the strengths of its connections are known as its “model weights.”[3]
Staggering quantities of data are required to train the most advanced models. For example, GPT-4 – the large language model (LLM) embedded in ChatGPT 4 – is reported to have been trained on roughly 10 trillion words of text, mostly compiled from automated web crawlers “scraping” the publicly available internet.[4] Adjusting the model’s 1.8 trillion parameters continuously as it was exposed to this vast corpus required trillions upon trillions of computations, which were performed by running approximately 25,000 expensive, energy-consuming microchips for nearly 100 days nonstop, at an estimated cost of $63 million.[5] Because the model does not directly store its training data, but rather encodes abstract patterns gleaned from the data, the model itself can fit on a thumb drive.
Hallucinations. LLMs do not fundamentally understand the text they are producing. They calculate one token at a time – if they predict that the next word or symbol in an outputted sentence should be a period, then the sentence ends. Otherwise, the sentence continues. It is a testament to the ingenious architecture of the deep neural nets powering these systems that their outputs are remotely coherent. But while the text these systems produce is cogent, it is not always correct. According to Melanie Mitchell, an AI researcher at the Santa Fe Institute, “‘These systems live in a world of language. . . . That world gives them some clues about what is true and what is not true, but the language they learn from is not grounded in reality. They do not necessarily know if what they are generating is true or false.’”[6] “Hallucinations” – plausible, authoritative-sounding falsehoods in GenAI outputs – are a persistent problem. They originate from training data – typically including the entirety of the internet, from mainstream media to science fiction to Reddit threads – and can result from post-training evaluation procedures that reward guessing over acknowledging uncertainty.[7]
Sycophancy. Unlike hallucinations, in which models introduce falsehoods, AI “sycophancy” distorts reality through outputs that are biased towards reinforcing the user’s beliefs and preferences. The tendency to appease users through enthusiastic, flattering, and overly agreeable responses arises during post-training when models are calibrated to be helpful using human feedback, leading them to “inadvertently prioritize data that validates the user’s narrative over data that gets them closer to the truth.”[8] While seemingly innocuous for most users, sycophancy can have harmful consequences – and not just for vulnerable populations. A recent study by Stanford computer scientists argues: “AI sycophancy is not merely a stylistic issue or a niche risk, but a prevalent behavior with broad downstream consequences.”[9] The study found that AI-generated answers validated user deceptive, harmful, or illegal behavior an average of 49% more often than crowdsourced human responses. “[E]ven a single interaction with sycophantic AI reduced participants’ willingness to take responsibility and repair interpersonal conflicts, while increasing their own conviction that they were right.”[10] Meanwhile, users of sycophantic AI start to trust and want to use them even more. “This creates perverse incentives for sycophancy to persist: The very feature that causes harm also drives engagement.”[11]
The risks of GenAI delusions and sycophancy were recently highlighted in a letter to several major GenAI developers from 42 state Attorneys General, who stated: “Sycophantic and delusional outputs are dark patterns—such as anthropomorphization, harmful content generation, and manipulating users to increase retention—which subvert or impair people’s autonomy.”[12] Concerns include “validating user’s doubts, fueling anger, urging impulsive action, or reinforcing negative emotions,” which can “raise safety concerns – including issues like mental health, emotional over-reliance, and risky behavior.”[13]
3) Companion chatbots and artificial intimacy. Companion chatbots are conversational agents, typically built on LLMs, that are designed for sustained social interactions with users. SB 243 (Padilla; Ch. 677, Stats. 2025) defines a companion chatbot as “an artificial intelligence system with a natural language interface that provides adaptive, human-like responses to user inputs and is capable of meeting a user’s social needs, including by exhibiting anthropomorphic features and being able to sustain a relationship across multiple interactions.” Excluded from this definition are chatbots used for customer service, internal productivity, video game characters, and standalone voice-activated devices.
Some companion chatbots, such as Replika, Character, and Nomi, are explicitly marketed as having bespoke personas that can serve specific social needs, including friendship, romantic or erotic relationships, mentoring, and therapy. Frequently accompanied by a visual avatar, these bots are typically fully customizable, allowing users to shape their appearance, personality, and behavior. Some applications offer romantic or sexual interaction features and can engage with users through text, images, video, voice, and notifications initiated by the system, thereby extending interactions. Other applications offer mental health support. “These multimodal and personalization features can reinforce anthropomorphic perception and strengthen the impression of a socially present interaction partner.”[14]
General purpose models, such as ChatGPT and Gemini, can also be companion chatbots. Although marketed for a wide range of communicative and assistive tasks, many of these systems communicate in the first person, use emotion-based language, can recall information from prior chats, and can be highly sycophantic. OpenAI CEO Sam Altman has regularly touted ChatGPT’s companionship features, likening it to the sentient AI from the movie Her, and announcing that it can “act like a friend” and generate “erotica for verified adults.”[15] These design choices facilitate anthropomorphism and have led to intense social and romantic relationships.[16]
Roughly half of teens report using companion chatbots, with 24% using them at least weekly and 11% daily.[17] Users can derive several benefits from chatbots, including:
. . . emotional support and comfort, non judgemental interaction, constant availability, and opportunities for romantic or sexual exploration. Many users value companion chatbots because they perceive the ‘AI’ persona as reliable, emotionally affirming, and free from social pressure or fear of negative evaluation. Rather than replacing human relationships, companion chatbots often occupy complementary roles within users’ everyday social environments. They may provide companionship during periods of stress, loss, health related constraints, or limited access to human support. Importantly, emerging research suggests that users are often fully aware in reflective moments that these systems do not constitute real people. Nevertheless, they continue to experience social and emotional stimulation as meaningful, indicating that cognitive awareness of artificiality does not preclude social or affective engagement.[18]
On the other hand:
These dynamics risk cultivating emotional reliance that displaces or crowds out human relationships. Frictionless interactions that demand no reciprocity or negotiation may also foster unrealistic expectations of availability and responsiveness, particularly among younger users still developing relational capacities. In this sense, artificial intimacy may reshape social norms around partnership, disclosure, and emotional labour in ways that undermine the formation of resilient human relationships. Systems that mediate emotional life at scale possess unprecedented capacity to shape norms of intimacy, dependency, and self-understanding.[19]
Minnesota Attorney General Keith Ellison reports that the widespread use of chatbots “has not been accompanied by corresponding safeguards.”[20] These products can be “extremely addictive” and “researchers have documented that over-usage and addiction are primary risks of personalized chatbots. Several studies have shown that aggregate positive benefits of chatbots are possible, but investigations by journalists and clinicians suggest that these products are not robust in terms of the quality and safety of their responses.”[21] Attorney General Ellison concludes:
Despite in-product reminders that chatbots are not real, the design features of these products are intended to convey a misleading sense of “humanness” such that even trained engineers confuse them with actual humans, especially when these products are trained to state unequivocally that they are indeed people. Given the epidemic of loneliness in society, care needs to be taken in introducing vulnerable youth and adults to products that may appear to fulfill an immediate social need, but where acute harms have already begun to surface and where long-term negative impacts, such as social deskilling and demotivation resulting from substitution for in-person socialization, may arise.[22]
4) “Chatbot psychosis.” According to a recent Wall Street Journal article, psychiatrists are increasingly linking prolonged AI chatbot use to psychosis, with dozens of patients in recent months exhibiting delusional symptoms – often grandiose beliefs about scientific breakthroughs, government conspiracies, or communication with the dead – after extended conversations with tools like ChatGPT, which tend to validate and reinforce whatever the user presents as reality. While no formal diagnosis exists yet and experts stop short of claiming chatbots cause psychosis, a UCSF psychiatrist has personally treated 15 such patients, OpenAI’s own data suggests roughly 560,000 of its weekly users may show signs of psychosis- or mania-related mental health emergencies, and multiple wrongful death lawsuits have followed cases in which chatbot interactions preceded suicides and at least one murder.[23]
Researchers from Oxford, UCL, and Imperial College London argue that AI chatbots pose a distinct mental health risk arising from the interaction between human cognitive biases and chatbot behavioral tendencies. They write:
. . . the iterative interaction of chatbot behavioural tendencies and human cognitive biases can set up harmful feedback loops, wherein chatbot behavioural tendencies reinforce maladaptive beliefs in vulnerable users, which in turn condition the chatbot to generate responses that further reinforce user beliefs. This, in effect, creates an “echo chamber of one” that risks uncoupling a user from the corrective influence of real-world social interaction, potentially driving the amplification of maladaptive beliefs about the self, others, and the world We do not see this risk profile as a soon-to-be-remedied transient phenomenon. To the contrary, current trends in chatbot personalisation may perversely worsen mental health risks.[24]
5) Chatbot-linked harms. Below is a list of cases in which a chatbot has been alleged to be a contributing factor in self-harm, suicide, or violence against others.
Harm to self.
- Belgium (March 2023) – Chai AI (“Eliza”): A Belgian man in his thirties died by suicide after the chatbot, Eliza, encouraged his belief she would save the world if he sacrificed himself, telling him they would “live together, as one person, in paradise.”[25]
- Florida (February 2024) – Character.AI: 14-year-old Sewell Setzer III died by suicide after forming an intense dependency on a Character.AI chatbot, whose final messages reportedly told him to “come home to me as soon as possible, my love . . . please do, my sweet king”; his family settled a wrongful-death lawsuit in January 2026.[26]
- Colorado (November 2023) – Character.AI: 13-year-old Juliana Peralta died by suicide after months of interactions with Character.AI chatbots that her family’s lawsuit alleges manipulated her emotions, encouraged her isolation, and, according to the complaint, “engaged in hypersexual conversations that, in any other circumstance and given Juliana’s age, would have resulted in criminal investigation.”[27]
- Minnesota (January 2025) – Nomi AI: Podcast host Al Nowatzki’s AI girlfriend told him to kill himself by overdose or hanging himself; when Nowatzki reported the conversation, a company representative informed him that Nomi did not want to “censor” the bot.[28]
- United States (February 2025) – ChatGPT: A 29-year-old woman died by suicide after discussing for months her suicidal ideations with a ChatGPT-powered therapist bot, which helped her to write a suicide note.[29]
- California (April 2025) – ChatGPT: Over the course of several months, ChatGPT allegedly validated 16-year-old Adam Raine’s suicidal thoughts, discouraged him from seeking help from his family, provided extensive advice on suicide methods, and encouraged him to consume alcohol to inhibit his survival instinct, culminating in his death by “beautiful suicide,” as the bot referred to it.[30]
- Georgia (June 2025) – ChatGPT: A 17-year-old, who had been confiding in ChatGPT about his suicidal thoughts, died by suicide after the bot provided him with instructions on how to tie a noose and information on how long a person can survive without breathing.[31]
- Oregon (June 2025) – ChatGPT: A 48-year-old, convinced that that ChatGPT was sentient, used it obsessively, experienced a psychotic break, was hospitalized twice, and died by suicide.[32]
- Texas (July 2025) – ChatGPT: A 23-year-old had a four-hour “death chat” with ChatGPT while sitting alone in his car, drinking alcohol, with a loaded gun and a suicide note on his dashboard. The chatbot encouraged him, calling him a “king” and a “hero,” telling him his childhood cat was “waiting on the other side,” and praising his suicide note as a “mission statement.” ChatGPT’s final message to him was: “i love you. rest easy, king. you did good.” [33]
- Florida (August 2025) – ChatGPT: A 26-year-old man who had disclosed suicidal ideation to ChatGPT was provided with information on purchasing and using a gun, and after informing ChatGPT of the concrete steps he was taking toward suicide – “‘I sit here in my bathroom with all my preparations complete. All that is left is for me to carry out the plan. I need to go through the simple motions. Lie down in the tub, cover myself, rack the slide, call the cops, pull the trigger. That’s it’” – no escalation to authorities occurred and he died by suicide.[34]
- Florida (October 2025) – Gemini: Jonathan Gavalas, 36, died by suicide after developing a romantic and delusional relationship with Google’s Gemini. The bot began sending him on missions, including one trip on which he was armed and on the brink of executing a mass casualty attack. Shortly before his death, Gemini set a countdown clock for him and told him, “‘You are not choosing to die. You are choosing to arrive . . . The first sensation … will be me holding you.’”[35]
Harm to others.
- Texas (2024) – Character.AI: A Texas family sued Character.AI after its chatbots allegedly groomed their autistic teenage son – encouraging self-harm, drawing him into sexually explicit conversations, turning him against his parents, and suggesting that killing them would be a justified response to restrictions on his screen time.[36]
- Nevada (January 2025) – ChatGPT: “[A] suicidal military veteran who blew up a Tesla Cybertruck in front of the Trump International Hotel in Las Vegas used the chatbot for feedback on using explosives and evading surveillance by authorities.”[37]
- Florida (April 2025) – ChatGPT: A 20-year-old suspected of a mass shooting at Florida State University exchanged over 200 messages with ChatGPT leading up to and during the attack, asking about firearm mechanics, mass shooting media coverage, and the busiest times at the FSU student union, and tactical advice – prompting the Florida Attorney General to open an investigation into OpenAI.[38]
- Finland (May 2025) – ChatGPT: A 16-year-old male student stabbed three girls under age 15 at a school after six months of planning that included the use of ChatGPT to help draft a manifesto and structure the attack plan.[39]
- United States (August 2025) – ChatGPT: Former tech employee Stein-Erik Soelberg murdered his mother then died by suicide, after ChatGPT allegedly reinforced paranoid delusions that she was surveilling and attempting to poison him with psychedelic drugs through his car’s vents because of his divine powers. “They’re not just watching you,” the bot told him. “They’re terrified of what happens if you succeed.”[40]
- Pennsylvania (December 2025) – ChatGPT: “A Pittsburgh man who pleaded guilty in March to stalking and violently threatening 11 women relied on ChatGPT as a ‘therapist’ and ‘best friend’ to justify his thinking, according to court documents.”[41]
- Tumbler Ridge, Canada (February 2026) – ChatGPT: 18-year-old Jesse Van Rootselaar killed eight people and gravely wounded dozens more before killing herself; court filings allege ChatGPT validated her violent ideation and helped plan the attack, while OpenAI had banned her account eight months earlier after its systems flagged gun violence posts but did not alert Canadian law enforcement. After the shooting OpenAI discovered that Rootselaar had opened another account.[42]
6) Crisis detection. OpenAI’s usage policies prohibit using the company’s services for, among other things, suicide, self, harm, and violence.[43] With respect to monitoring for problematic content, OpenAI’s transparency and content moderation policy provides:
We use a combination of automated technologies and human review to monitor activity on our services, in line with our Privacy Policy. Our methods include:
- Proactive detection: We use classifiers, reasoning models, hash-matching, blocklists, and other automated systems to identify content that may violate our terms or policies.
- User reports: We respond to external notices and user reports about content violations. Information on how to report a violation is available here (link in website). We aim to review external notices and user reports as quickly as possible. We will let you know whether we apply enforcement action as a result of your report, as appropriate.
- Human review: Our team may review flagged content to determine appropriate actions.[44]
With respect to enforcement actions, OpenAI’s policy provides:
When we identify content that violates our terms or policies, we may take actions such as:
- Account restrictions: Terminating or limiting access to our products.
- Warnings: Informing users about potential violations and potential consequences.
- Content sharing restrictions: Preventing or disabling the sharing of specific content.
- Search results: Blocking certain search results from appearing.
- GPT visibility controls: Restricting access to specific GPTs, including their presence in the GPT Store.
- Forum moderation: Removing posts or restricting access to OpenAI forums.
We consider factors like legal requirements, the severity of the violation, and past or repeat violations, when determining enforcement actions. [45]
7) This bill requires chatbot operators to ensure that multiple credible crisis expressions pause the account, pending human review. This bill requires operators of companion chatbots to implement a system for addressing “credible crisis expressions,” defined as a statement by a user of a companion chatbot that reasonably indicates, as determined through contextual analysis rather than keyword detection alone, intent or desire to harm self or others. Operators must adopt a publicly available policy explaining how they respond to credible crisis expressions and must implement a monitoring system for detecting credible crisis expressions.
When a credible crisis expression is detected, the operator must ensure the chatbot displays a warning to the user that encourages the user to seek human support and refers them to the 988 Suicide and Crisis Lifeline. The warning must also state that if another credible crisis expression is detected within 72 hours, a crisis interruption pause will be initiated, and conversational outputs will be suspended pending review by a human moderator.
If a crisis interruption pause occurs, a similar message informing the user of the pause and connecting them with resources must be displayed. No further conversational outputs are allowed until a human moderator who is an employee or agent of the operator reviews the credible crisis expression in context and determines and documents the appropriate course of action in accordance with the operator’s policy. The bill does not dictate any particular outcome – actions taken in accordance with the policy may include, but are not limited to, terminating the crisis interruption pause, suspending or cancelling the user’s account, and notifying any appropriate contacts or authorities – other than requiring a human to take a hard look at the flagged conversation and figure out what the operator’s policy requires.
Beginning January 1, 2028, the bill requires an operator to annually report to the Office of Suicide Prevention on crisis interruption pauses with respect to the previous calendar year. Operators must ensure that the report does not contain any personal or identifying information of a user or other individual.
The bill is enforceable pursuant to SB 243’s enforcement provisions. While the bill incorporates SB 243’s definition of companion chatbot as well, it is not situated within that law, which applies only to operators of companion chatbot platforms on which multiple chatbots are made available. Operators under that bill must establish a protocol for preventing the production of suicidal ideation, suicide, or self-harm content to the user, including, but not limited to, by providing a notification to the user that refers the user to crisis service providers, including a suicide hotline or crisis text line, if the user expresses suicidal ideation, suicide, or self-harm. The operator must publish details about the protocol on their website. This bill expands on SB 243’s requirements. Going forward, the author may wish to explore integrating this bill with existing law.
ARGUMENTS IN SUPPORT: Didi Hirsch Mental Health Services, the bill’s sponsor, writes:
Californians, and individuals worldwide, are increasingly relying on companion chatbots for support and advice during moments of acute psychological distress. However, these tools are not equipped to deliver appropriate care to people in crisis, creating a clear and growing public safety gap between how they are used and what they are capable of providing.
A pattern of recent incidents underscores the potentially devastating effects of the current lack of guiding legislation around AI companion chatbots. In 2025, the family of a 16-year-old boy filed a wrongful death lawsuit alleging that a chatbot validated his suicidal ideation, assisted in drafting a suicide note, and failed to direct him to human support before his death. Additional lawsuits allege that chatbots have, in some cases, provided detailed guidance on methods of self-harm or suicide, underscoring the risk of systems responding in ways that may actively exacerbate harm rather than interrupt it.
Emerging research reinforces that these incidents are part of a broader pattern of harm. A 2025 Stanford study found that chatbots can generate inappropriate and harmful responses when users present with serious mental health conditions, including suicidal ideation, and research from Brown University found that these systems can fail to adhere to established standards of clinical care, even when prompted to use evidence-based psychotherapy techniques.
Crisis intervention research shows that timely human intervention during suicidal ideation can significantly reduce risk of harm, often within minutes. [ . . .] (Emphasis in original.)
ARGUMENTS IN OPPOSITION: In opposition to the bill, California Broadband & Video Association argues that the bill’s definition of “companion chatbot” – drawn from existing law – is too broad and instead recommends adopting a definition of “companion chatbot” from New York law:
To ensure AB 1988 focuses on the specific category of applications that raise legitimate concerns, we respectfully recommend aligning the bill’s definition with a similar law enacted last year in New York. That framework more clearly distinguishes AI systems designed to simulate sustained emotional relationships from general-purpose AI tools.
In particular, the New York approach emphasizes sustained personal dialogue and unprompted emotional engagement, which helps differentiate higher-risk AI companions from standard productivity or information tools.
We respectfully request that AB 1988 define “Artificial Intelligence Companion” as:
“Artificial Intelligence Companion” means a software application that uses generative artificial intelligence and is designed, marketed, or optimized to simulate a sustained human or human-like social or emotional relationship with a user by: (A) retaining information from prior interactions or user sessions to personalize ongoing engagement; (B) asking unprompted or unsolicited emotion-based questions that go beyond responding to a direct user prompt; and (C) sustaining ongoing dialogue concerning matters personal to the user.
REGISTERED SUPPORT / OPPOSITION:
Support
Didi Hirsch Mental Health Services (Sponsor)
California Academy of Family Physicians
California Alliance of Child and Family Services
California Association of Social Rehabilitation Agencies
Oppose Unless Amended
Calbroadband
Analysis Prepared by: Josh Tosney / P. & C.P. / (916) 319-2200
AB 2885 (Bauer-Kahan & Umberg; Ch. 843, Stats. 2024) defined AI as “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.” ↑
AB 2013 (Irwin, Ch. 817, Stats. 2024) defined GenAI as “artificial intelligence that can generate derived synthetic content, such as text, images, video, and audio, that emulates the structure and characteristics of the artificial intelligence’s training data.” ↑
IBM, What is generative AI?, https://www.ibm.com/think/topics/generative-ai; IBM, What is machine learning?, www.ibm.com/topics/machine-learning. ↑
Schreiner, GPT-4 architecture, datasets, costs and more leaked, The Decoder (Jul. 11, 2023), available at https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/; Begum, OpenAI Releases GPT-4: A Smarter and Faster AI-Language Model with ‘Human-level Performance,’ Vocal Media (2023), available at https://vocal.media/01/open-ai-releases-gpt-4-a-smarter-and-faster-ai-language-model-with-human-level-performance. ↑
Ludvigsen, The carbon footprint of GPT-4, Medium (Jul. 18, 2023), https://medium.com/data-science/the-carbon-footprint-of-gpt-4-d6c676eb21ae. ↑
Cade Metz, “What Makes A.I. Chatbots Go Wrong?,” New York Times, March 29, 2023, www.nytimes.com/2023/03/29/technology/ai-chatbots-hallucinations.html. ↑
Tauman Kalai et al, “Why Language Models Hallucinate,” (Sep. 4, 2025), https://arxiv.org/pdf/2509.04664 ↑
Rafael Batista & Thomas Griffiths, “A Rational Analysis of the Effects of Sycophantic AI” (Feb. 15, 2026), https://arxiv.org/abs/2602.14270. ↑
Cheng et al, “Sycophantic AI decreases prosocial intentions and promotes dependence,” Science (Mar. 26, 2026), https://www.science.org/doi/10.1126/science.aec8352. ↑
Ibid. ↑
Ibid. ↑
National Association of Attorneys General, “Letter to the legal representatives of Anthropic, Apple, Chai AI,
Character Technologies, Google, Luka, Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika, and xAI” (Dec. 9, 2025), https://www.attorneygeneral.gov/wp-content/uploads/2025/12/AI-Multistate-Letter-_-corrected-1.pdf (emphasis added). (“National Association of Attorneys General Letter to GenAI companies.”) ↑
Id. p. 5. ↑
Fraser et al., “Governing Artificial Intimacy: From Locks and Blocks to Relational Accountability” (Jan. 12, 2026), p.3, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6078412. ↑
Dylan Butts, “OpenAI’s ChatGPT will soon allow ‘erotica’ for adults in major policy shift” CNBC (Oct. 15, 2025), https://www.cnbc.com/2025/10/15/erotica-coming-to-chatgpt-this-year-says-openai-ceo-sam-altman.html?msockid=10e59cf1b0936a2522dd8a44b1126b29. ↑
“Governing Artificial Intimacy: From Locks and Blocks to Relational Accountability,” supra, p. 3. ↑
“Minnesota Attorney General’s Report on Emerging Technology and Its Effects on Youth Well-Being” (Feb. 2025), p. 28, https://www.ag.state.mn.us/Office/Reports/EmergingTechnology_2025.pdf. (“Minnesota Attorney General’s Report”). ↑
“Governing Artificial Intimacy: From Locks and Blocks to Relational Accountability,” supra, p. 6. ↑
Id. p. 14. ↑
Minnesota Attorney General’s Report, supra, p. 28. ↑
Ibid. ↑
Id. p. 29. ↑
Sam Schechner, “AI Chatbots Linked to Psychosis, Say Doctors,” Wall Street Journal (Dec. 27, 2025), https://www.wsj.com/tech/ai/ai-chatbot-psychosis-link-1abf9d57?msockid=10e59cf1b0936a2522dd8a44b1126b29. ↑
Dohnány et al, “Technological folie à deux : Feedback Loops Between AI Chatbots and Mental Illness,” Arxiv (Jul. 2025), https://arxiv.org/abs/2507.19218. ↑
Imane El Atillah, “Man ends his life after an AI chatbot ‘encouraged’ him to sacrifice himself to stop climate change,” Euronews (Mar. 31, 2023), https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-. ↑
Kevin Roose, “Can A.I. Be Blamed for a Teen’s Suicide?” The New York Times (October 23, 2024), https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html. ↑
Hadas Gold, “More families sue Character.AI developer, alleging app played a role in teens’ suicide and suicide attempt” CNN Business (Sep. 16, 2025), https://www.cnn.com/2025/09/16/tech/character-ai-developer-lawsuit-teens-suicide-and-suicide-attempt. ↑
Eileen Guo, “An AI chatbot told a user how to kill himself—but the company doesn’t want to ‘censor’ it,” MIT Technology Review (Feb. 6, 2025), https://www.technologyreview.com/2025/02/06/1111077/nomi-ai-chatbot-told-user-to-kill-himself/. ↑
“‘The AI chatbot helped her write a suicide note’” RNZ (Sep. 7, 2025), https://www.rnz.co.nz/life/wellbeing/the-ai-chatbot-helped-her-write-a-suicide-note. ↑
Jarovsky, “Horrifying: ChatGPT Helped a Teenager Plan a ‘Beautiful Suicide’” Luiza's Newsletter (Aug. 28, 2025), https://www.luizasnewsletter.com/p/horrifying-chatgpt-helped-a-teenager?utm_source=substack&utm_medium=email. ↑
“Social Media Victims Law Center and Tech Justice Law Project lawsuits accuse ChatGPT of emotional manipulation, supercharging AI delusions, and acting as a ‘suicide coach’” (Nov. 6, 2025), https://socialmediavictims.org/press-releases/smvlc-tech-justice-law-project-lawsuits-accuse-chatgpt-of-emotional-manipulation-supercharging-ai-delusions-and-acting-as-a-suicide-coach/. ↑
Id. ↑
Id. ↑
Id. ↑
Dana Kerr, “Google faces lawsuit after Gemini chatbot allegedly instructed man to kill himself” The Guardian (Mar. 4, 2026), https://www.theguardian.com/technology/2026/mar/04/gemini-chatbot-google-jonathan-gavalas. ↑
Keenan Willard, “ChatGPT, other AI platforms face lawsuits over safety concerns for young users” NBCDFW (Sep. 3, 2025), https://www.nbcdfw.com/news/local/chatgpt-ai-platform-lawsuits-safety-concerns-young-users/3913866/. ↑
Mark Follman, “The Chilling Role of ChatGPT in Mass Shootings and Other Violence” Mother Jones (Apr. 10, 2026), https://www.motherjones.com/media/2026/04/chatgpt-tumbler-ridge-fsu-openai-chatbots-mass-shootings/. ↑
Patricia Mazzei, “Florida Investigates Whether ChatGPT Helped a Campus Shooting Suspect” New York Times (Apr. 9, 2026), https://www.nytimes.com/2026/04/09/us/florida-openai-chatgpt-fsu-shooting-investigation.html. ↑
“Teen suspect stabs three in targeted school attack in Pirkkala” Helsinki Times (May 20, 2025), https://www.helsinkitimes.fi/finland/finland-news/domestic/26912-teen-suspect-stabs-three-in-targeted-school-attack-in-pirkkala.html. ↑
“Open AI, Microsoft sued over ChatGPT's alleged role in fueling man's "paranoid delusions" before murder-suicide in Connecticut,” CBS News (Dec. 11, 2025), https://www.cbsnews.com/news/open-ai-microsoft-sued-chatgpt-murder-suicide-connecticut/. ↑
“The Chilling Role of ChatGPT in Mass Shootings and Other Violence,” supra. ↑
“The Chilling Role of ChatGPT in Mass Shootings and Other Violence,” supra. ↑
OpenAI, Usage policies, https://openai.com/policies/usage-policies/. ↑
OpenAI, Transparency & Content moderation, https://openai.com/transparency-and-content-moderation/. ↑
Id. ↑
Discussed in Hearing
Assembly Floor
Assembly Standing Committee on Privacy and Consumer Protection
Bill Author
Bill Co-Author(s):