AB 3030: Health care services: artificial intelligence.
- Session Year: 2023-2024
- House: Assembly
- Latest Version Date: 2024-09-28
Current Status:
Passed
(2024-09-28: Chaptered by Secretary of State - Chapter 848, Statutes of 2024.)
Introduced
In Committee
First Chamber
In Committee
Second Chamber
Enacted
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. Existing law, the Medical Practice Act, establishes the Medical Board of California for the licensing, regulation, and discipline of physicians and surgeons. Existing law, the Osteopathic Act, enacted by an initiative measure, establishes the Osteopathic Medical Board of California for the licensing and regulation of osteopathic physicians and surgeons.
This bill would require a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. The bill would exempt from this requirement a communication read and reviewed by a human licensed or certified health care provider. Under the bill, a violation of these provisions by a physician would be subject to the jurisdiction of the Medical Board of California or Osteopathic Medical Board of California, as appropriate.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. Existing law, the Medical Practice Act, establishes the Medical Board of California for the licensing, regulation, and discipline of physicians and surgeons. Existing law, the Osteopathic Act, enacted by an initiative measure, establishes the Osteopathic Medical Board of California for the licensing and regulation of osteopathic physicians and surgeons.
This bill would require a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. The bill would exempt from this requirement a communication read and reviewed by a human licensed or certified health care provider. Under the bill, a violation of these provisions by a physician would be subject to the jurisdiction of the Medical Board of California or Osteopathic Medical Board of California, as appropriate.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. Existing law, the Medical Practice Act, establishes the Medical Board of California for the licensing, regulation, and discipline of physicians and surgeons. Existing law, the Osteopathic Act, enacted by an initiative measure, establishes the Osteopathic Medical Board of California for the licensing and regulation of osteopathic physicians and surgeons.
This bill would require a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, as specified. The bill would exempt from this requirement a communication read and reviewed by a human licensed or certified health care provider. Under the bill, a violation of these provisions by a physician would be subject to the jurisdiction of the Medical Board of California or Osteopathic Medical Board of California, as appropriate.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. A violation of these provisions is a crime. Existing law, the Medical Practice Act, establishes the Medical Board of California for the licensing, regulation, and discipline of physicians and surgeons. Existing law, the Osteopathic Act, enacted by an initiative measure, establishes the Osteopathic Medical Board of California for the licensing and regulation of osteopathic physicians and surgeons.
This bill would require a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, as specified. Because a violation of these requirements would be a crime, this bill would impose a state-mandated local program. The bill would exempt from this requirement a communication read and reviewed by a human licensed or certified health care provider. Under the bill, a violation of these provisions by a physician would be subject to the jurisdiction of the Medical Board of California or Osteopathic Medical Board of California, as appropriate.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.This bill would provide that no reimbursement is required by this act for a specified reason.Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. A violation of these provisions is a crime.
This bill would require a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions permitting a patient to communicate with a human health care provider. describing how a patient may contact a human health care provider, as specified. Because a violation of these requirements would be a crime, this bill would impose a state-mandated local program.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. A violation of these provisions is a crime.
This bill would require a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions permitting a patient to communicate with a human health care provider. Because a violation of these requirements would be a crime, this bill would impose a state-mandated local program.
The California Constitution requires the state to reimburse local agencies and school districts for certain costs mandated by the state. Statutory provisions establish procedures for making that reimbursement.
This bill would provide that no reimbursement is required by this act for a specified reason.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. A violation of these provisions is a crime.This bill would require an entity, including a health facility, clinic, physicians office, or office of a group practice that uses a generative artificial intelligence tool to generate responses for health care providers to communicate with patients to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by artificial intelligence and (2) clear instructions for the patient to access direct communications with a health care provider, as specified. The bill would prohibit an entity or health care provider from being subject to any disciplinary action related to licensure or certification solely because of the entitys or health care providers failure to comply with these provisions.Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. A violation of these provisions is a crime.
This bill would require an entity, including a health facility, clinic, physicians office, or office of a group practice that uses a generative artificial intelligence tool to generate responses for health care providers to communicate with patients to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by artificial intelligence and (2) clear instructions for the patient to access direct communications with a health care provider, as specified. The bill would prohibit an entity or health care provider who fails to comply with these provisions from being subject to any disciplinary action related to licensure or certification, or to any civil or criminal liability for that failure. certification solely because of the entitys or health care providers failure to comply with these provisions.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. A violation of these provisions is a crime.
This bill would require an entity, including a health facility, clinic, physicians office, or office of a group practice that uses a generative artificial intelligence tool to generate responses for health care providers to communicate with patients to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by artificial intelligence and (2) clear instructions for the patient to access direct communications with a health care provider, as specified. The bill would prohibit an entity or health care provider who fails to comply with these provisions from being subject to any disciplinary action related to licensure or certification, or to any civil or criminal liability for that failure.
Existing law sets forth the powers and duties of the protection and advocacy agency, which is a private nonprofit corporation designated by the Governor pursuant to federal law for the protection and advocacy of the rights of persons with disabilities.Existing law requires the agency to meet all of the federal requirements applicable to protection and advocacy systems, including, but not limited to, the requirement that it establish a grievance procedure for clients or prospective clients of the system to ensure that people with disabilities have full access to services of the system. Existing law requires the agency to cooperate with the Office of the State Long-Term Care Ombudsman when appropriate.This bill would make technical, nonsubstantive changes to related provisions.Existing law sets forth the powers and duties of the protection and advocacy agency, which is a private nonprofit corporation designated by the Governor pursuant to federal law for the protection and advocacy of the rights of persons with disabilities.
Existing law requires the agency to meet all of the federal requirements applicable to protection and advocacy systems, including, but not limited to, the requirement that it establish a grievance procedure for clients or prospective clients of the system to ensure that people with disabilities have full access to services of the system. Existing law requires the agency to cooperate with the Office of the State Long-Term Care Ombudsman when appropriate.
This bill would make technical, nonsubstantive changes to related provisions.
(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information shall ensure that those communications include both of the following:
(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.
(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2)Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physicians office, or office of a group provider, or other appropriate person.
(b)If a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirements of subdivision (a) do not apply.
(c)For purposes of this section, the following definitions apply:
(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(2)Clinic has the same meaning as defined in Section 1200.
(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Health facility has the same meaning as defined in Section 1250.
(6)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(7)Patient clinical information means information relating to the health status of a patient. This information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters.
(8)Physicians office means an office of a physician in solo practice.
(d)(1)A violation of this section by a licensed health facility is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1275) of Chapter 2.
(2)A violation of this section by a licensed clinic is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1225) of Chapter 1.
(3)A violation of this section by a physician is subject to the jurisdiction of the Medical Board of California or the Osteopathic Medical Board of California, as appropriate.
(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information shall ensure that those communications include both of the following:
(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.
(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2)Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physicians office, or office of a group provider, or other appropriate person.
(b)If a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirements of subdivision (a) do not apply.
(c)For purposes of this section, the following definitions apply:
(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(2)Clinic has the same meaning as defined in Section 1200.
(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Health facility has the same meaning as defined in Section 1250.
(6)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(7)Patient clinical information means information relating to the health status of a patient. This information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters.
(8)Physicians office means an office of a physician in solo practice.
(d)(1)A violation of this section by a licensed health facility is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1275) of Chapter 2.
(2)A violation of this section by a licensed clinic is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1225) of Chapter 1.
(3)A violation of this section by a physician is subject to the jurisdiction of the Medical Board of California or the Osteopathic Medical Board of California, as appropriate.
(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information shall ensure that those communications include both of the following:
(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.
(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2)Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physicians office, or office of a group provider, or other appropriate person.
(b)If a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirements of subdivision (a) do not apply.
(c)For purposes of this section, the following definitions apply:
(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(2)Clinic has the same meaning as defined in Section 1200.
(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Health facility has the same meaning as defined in Section 1250.
(6)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(7)Patient clinical information means information relating to the health status of a patient. This information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters.
(8)Physicians office means an office of a physician in solo practice.
(d)(1)A violation of this section by a licensed health care facility is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1275) of Chapter 2.
(2)A violation of this section by a licensed clinic is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1225) of Chapter 1.
(3)A violation of this section by a physician is subject to the jurisdiction of the Medical Board of California or the Osteopathic Medical Board of California, as appropriate.
(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information shall ensure that those communications include both of the following:
(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.
(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2)Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physicians office, or office of a group provider, or other appropriate person.
(b)If a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirements of subdivision (a) do not apply.
(c)For purposes of this section, the following definitions apply:
(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(2)Clinic has the same meaning as defined in Section 1200.
(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Health facility has the same meaning as defined in Section 1250.
(6)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(7)Patient clinical information means information relating to the health status of a patient. This information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters.
(8)Physicians office means an office of a physician in solo practice.
(d)(1)A violation of this section by a licensed health care facility is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1275) of Chapter 2.
(2)A violation of this section by a licensed clinic is subject to the enforcement mechanisms described in Article 3 (commencing with Section 1225) of Chapter 1.
(3)A violation of this section by a physician is subject to the jurisdiction of the Medical Board of California or the Osteopathic Medical Board of California, as appropriate.
SECTION 1.Section 1316.9 is added to the Health and Safety Code, to read:1316.9.(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information shall ensure that those communications include both of the following:(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.(2)Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physicians office, or office of a group provider, or other appropriate person.(b)If a written communication is generated by generative artificial intelligence and reviewed by a human health care provider, the disclaimer shall indicate that the communication was generated by generative artificial intelligence and reviewed by a human.(c)For purposes of this section, the following definitions apply:(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.(2)Clinic has the same meaning as defined in Section 1200.(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.(5)Health facility has the same meaning as defined in Section 1250.(6)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.(7)Patient clinical information means information relating to the health status of a patient. This information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters.(8)Physicians office means an office of a physician in solo practice.SEC. 2.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information shall ensure that those communications include both of the following:
(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.
(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2)Clear instructions permitting a patient to communicate with a human health care provider. describing how a patient may contact a human health care provider, employee of the health facility, clinic, physicians office, or office of a group provider, or other appropriate person.
(b)If a written communication is generated by generative artificial intelligence and reviewed by a human health care provider, the disclaimer shall indicate that the communication was generated by generative artificial intelligence and reviewed by a human.
(c)For purposes of this section, the following definitions apply:
(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(2)Clinic has the same meaning as defined in Section 1200.
(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Health facility has the same meaning as defined in Section 1250.
(5)(6)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(7)Patient clinical information means information relating to the health status of a patient. This information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters.
(6)(8)Physicians office means an office of a physician in solo practice.
SEC. 2.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
(a)A health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications shall ensure that those communications include both of the following:
(1)A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(A)For written communications involving physical and digital media, including letters, emails, and other occasional messages, the disclaimer shall appear prominently at the beginning of each communication.
(B)For written communications involving continuous online interactions, including chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(C)For audio communications, the disclaimer shall be provided verbally at the start and the end of the interaction.
(D)For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2)Clear instructions permitting a patient to communicate with a human health care provider.
(b)If a written communication is generated by generative artificial intelligence and reviewed by a human health care provider, the disclaimer shall indicate that the communication was generated by generative artificial intelligence and reviewed by a human.
(c)For purposes of this section, the following definitions apply:
(1)Artificial intelligence means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(2)Clinic has the same meaning as defined in Section 1200.
(3)Generative artificial intelligence means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(6)Physicians office means an office of a physician in solo practice.
SEC. 2.No reimbursement is required by this act pursuant to Section 6 of Article XIIIB of the California Constitution because the only costs that may be incurred by a local agency or school district will be incurred because this act creates a new crime or infraction, eliminates a crime or infraction, or changes the penalty for a crime or infraction, within the meaning of Section 17556 of the Government Code, or changes the definition of a crime within the meaning of Section 6 of Article XIIIB of the California Constitution.
SECTION 1.Section 1316.9 is added to the Health and Safety Code, to read:1316.9.(a)An entity, including a health facility, clinic, physicians office, or office of a group practice that uses a generative artificial intelligence tool to generate responses for health care providers to communicate with patients through an online interface or via telephone shall ensure that those communications to patients include both of the following:(1)A disclaimer that indicates to the patient that a communication was generated by artificial intelligence.(2)Clear instructions for the patient to navigate the entitys internet website or other platforms to communicate directly with a health care provider. For purposes of this paragraph, communicate directly means to communicate without using responses generated by artificial intelligence.(b)If a response communicated through an online interface is reviewed by a human health care provider, the provider may indicate that the response was generated by artificial intelligence and reviewed by a human.(c)An entity or health care provider that fails to comply with the requirements of this section shall not be subject to any disciplinary actions related to licensure or certification solely because of the entitys or health care providers failure to comply with the requirements of this section.(d)Section 1290 does not apply to a violation of this section. (e)For purposes of this section, the following definitions apply: (1)Clinic has the same meaning as defined in Section 1200.(2)Generative artificial intelligence tool means a tool that uses machine learning systems capable of generating text, images, code, or other types of content, often in response to a prompt entered by a user. (3)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.(4)Physicians office means an office of a physician in solo practice.(a)An entity, including a health facility, clinic, physicians office, or office of a group practice that uses a generative artificial intelligence tool to generate responses for health care providers to communicate with patients through an online interface or via telephone shall ensure that those communications to patients include both of the following:
(1)A disclaimer that indicates to the patient that a communication was generated by artificial intelligence.
(2)Clear instructions for the patient to navigate the entitys internet website or other platforms to communicate directly with a health care provider. For purposes of this paragraph, communicate directly means to communicate without using responses generated by artificial intelligence.
(b)If a response communicated through an online interface is reviewed by a human health care provider, the provider may indicate that the response was generated by artificial intelligence and reviewed by a human.
(c)An entity or health care provider that fails to comply with the requirements of this section shall not be subject to any disciplinary actions related to licensure or certification, or to any civil or criminal liability, certification solely because of the entitys or health care providers failure to comply with the requirements of this section.
(d)Section 1290 does not apply to a violation of this section.
(e)For purposes of this section, the following definitions apply:
(1)Clinic has the same meaning as defined in Section 1200.
(2)Generative artificial intelligence tool means a tool that uses machine learning systems capable of generating text, images, code, or other types of content, often in response to a prompt entered by a user.
(3)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(4)Physicians office means an office of a physician in solo practice.
(a)An entity, including a health facility, clinic, physicians office, or office of a group practice that uses a generative artificial intelligence tool to generate responses for health care providers to communicate with patients through an online interface or via telephone shall ensure that those communications to patients include both of the following:
(1)A disclaimer that indicates to the patient that a communication was generated by artificial intelligence.
(2)Clear instructions for the patient to navigate the entitys internet website or other platforms to communicate directly with a health care provider. For purposes of this paragraph, communicate directly means to communicate without using responses generated by artificial intelligence.
(b)If a response communicated through an online interface is reviewed by a human health care provider, the provider may indicate that the response was generated by artificial intelligence and reviewed by a human.
(c)An entity or health care provider that fails to comply with the requirements of this section shall not be subject to any disciplinary actions related to licensure or certification, or to any civil or criminal liability, because of the entitys or health care providers failure to comply with the requirements of this section.
(d)Section 1290 does not apply to a violation of this section.
(e)For purposes of this section, the following definitions apply:
(1)Clinic has the same meaning as defined in Section 1200.
(2)Generative artificial intelligence tool means a tool that uses machine learning systems capable of generating text, images, code, or other types of content, often in response to a prompt entered by a user.
(3)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(4)Physicians office means an office of a physician in solo practice.
SECTION 1.Section 4901 of the Welfare and Institutions Code is amended to read:4901.(a)The protection and advocacy agency, for purposes of this division, shall be a private nonprofit corporation and shall meet all of the requirements of federal law applicable to protection and advocacy systems, including, but not limited to, the requirement that it establish a grievance procedure for clients or prospective clients of the system to ensure that people with disabilities have full access to services of the system.(b)State officers and employees, in taking any action relating to the protection and advocacy agency, shall meet the requirements of federal law applicable to protection and advocacy systems.(c)The authority of the protection and advocacy agency set forth in this division shall not diminish the authority of the protection and advocacy agency under federal statutes pertaining to the authority of protection and advocacy systems, or under federal rules and regulations adopted in implementation of those statutes.(d)This division shall not be construed to supplant the jurisdiction or the responsibilities of adult protective services programs pursuant to Chapter 11 (commencing with Section 15600), or Chapter 13 (commencing with Section 15750), of Part 3 of Division 9.(e)(1)This division shall not be construed to supplant the duties or authority of the State Long-Term Care Ombudsman Program pursuant to Chapter 11 (commencing with Section 9700) of Division 8.5.(2)The protection and advocacy agency shall cooperate with the Office of the State Long-Term Care Ombudsman when appropriate, as provided in Section 9717.(f)(1)This division shall not be construed to alter or impact the Elder and Dependent Adult Civil Protection Act (Chapter 11 (commencing with Section 15600) of Part 3 of Division 9), or Chapter 13 (commencing with Section 15750) of Part 3 of Division 9, including the confidentiality requirements of Section 15633 and the legal responsibility of the protection and advocacy agency to report elder or dependent adult abuse or neglect as required by paragraph (1) of subdivision (b) of Section 15630.(2)The adult protective services agency shall retain the responsibility to investigate any report of abuse or neglect in accordance with Chapter 13 (commencing with Section 15750) of Part 3 of Division 9 when the reported abuse or neglect is within the jurisdiction of the adult protective services agency.(a)The protection and advocacy agency, for purposes of this division, shall be a private nonprofit corporation and shall meet all of the requirements of federal law applicable to protection and advocacy systems, including, but not limited to, the requirement that it establish a grievance procedure for clients or prospective clients of the system to ensure that people with disabilities have full access to services of the system.
(b)State officers and employees, in taking any action relating to the protection and advocacy agency, shall meet the requirements of federal law applicable to protection and advocacy systems.
(c)The authority of the protection and advocacy agency set forth in this division shall not diminish the authority of the protection and advocacy agency under federal statutes pertaining to the authority of protection and advocacy systems, or under federal rules and regulations adopted in implementation of those statutes.
(d)Nothing in this division shall This division shall not be construed to supplant the jurisdiction or the responsibilities of adult protective services programs pursuant to Chapter 11 (commencing with Section 15600), or Chapter 13 (commencing with Section 15750), of Part 3 of Division 9.
(e)(1)Nothing in this division shall This division shall not be construed to supplant the duties or authority of the State Long-Term Care Ombudsman Program pursuant to Chapter 11 (commencing with Section 9700) of Division 8.5.
(2)The protection and advocacy agency shall cooperate with the Office of the State Long-Term Care Ombudsman when appropriate, as provided in Section 9717.
(f)(1)Nothing in this division shall This division shall not be construed to alter or impact the Elder and Dependent Adult Civil Protection Act (Chapter 11 (commencing with Section 15600), 15600) of Part 3 of Division 9), or Chapter 13 (commencing with Section 15750), 15750) of Part 3 of Division 9, including the confidentiality requirements of Section 15633 and the legal responsibility of the protection and advocacy agency to report elder or dependent adult abuse or neglect as required by paragraph (1) of subdivision (b) of Section 15630.
(2)The adult protective services agency shall retain the responsibility to investigate any report of abuse or neglect in accordance with Chapter 13 (commencing with Section 15750) of Part 3 of Division 9 when the reported abuse or neglect is within the jurisdiction of the adult protective services agency.
|
Date
|
Status
|
|---|---|
| Sep 28, 2024 | Chaptered by Secretary of State - Chapter 848, Statutes of 2024. |
| Sep 28, 2024 | Approved by the Governor. |
| Sep 13, 2024 | Enrolled and presented to the Governor at 4 p.m. |
| Aug 31, 2024 | Senate amendments concurred in. To Engrossing and Enrolling. (Ayes 75. Noes 0.). |
| Aug 30, 2024 | In Assembly. Concurrence in Senate amendments pending. |
| Aug 30, 2024 | Read third time. Passed. Ordered to the Assembly. (Ayes 40. Noes 0.). |
| Aug 26, 2024 | Ordered to special consent calendar. |
| Aug 19, 2024 | Read second time. Ordered to third reading. |
| Aug 15, 2024 | Read second time and amended. Ordered returned to second reading. |
| Aug 15, 2024 | From committee: Amend, and do pass as amended. (Ayes 7. Noes 0.) (August 15). |
| Aug 5, 2024 | In committee: Referred to APPR suspense file. |
| Jun 27, 2024 | Read second time and amended. Re-referred to Com. on APPR. |
| Jun 26, 2024 | From committee: Amend, and do pass as amended and re-refer to Com. on APPR. (Ayes 11. Noes 0.) (June 26). |
| Jun 18, 2024 | From committee chair, with author's amendments: Amend, and re-refer to committee. Read second time, amended, and re-referred to Com. on HEALTH. |
| May 29, 2024 | Referred to Com. on HEALTH. |
| May 21, 2024 | In Senate. Read first time. To Com. on RLS. for assignment. |
| May 20, 2024 | Read third time. Passed. Ordered to the Senate. (Ayes 71. Noes 0. Page 5321.) |
| May 13, 2024 | Ordered to third reading. |
| May 13, 2024 | From Consent Calendar. |
| May 9, 2024 | Read second time. Ordered to Consent Calendar. |
| May 8, 2024 | From committee: Do pass. To Consent Calendar. (Ayes 15. Noes 0.) (May 8). |
| Apr 29, 2024 | Re-referred to Com. on APPR. |
| Apr 25, 2024 | Read second time and amended. |
| Apr 24, 2024 | From committee: Amend, and do pass as amended and re-refer to Com. on APPR. (Ayes 10. Noes 0.) (April 23). |
| Apr 15, 2024 | Re-referred to Com. on P. & C.P. |
| Apr 11, 2024 | Read second time and amended. |
| Apr 10, 2024 | From committee: Amend, and do pass as amended and re-refer to Com. on P. & C.P. (Ayes 16. Noes 0.) (April 9). |
| Apr 1, 2024 | Re-referred to Com. on HEALTH. |
| Mar 21, 2024 | From committee chair, with author's amendments: Amend, and re-refer to Com. on HEALTH. Read second time and amended. |
| Mar 21, 2024 | Referred to Coms. on HEALTH and P. & C.P. |
| Feb 17, 2024 | From printer. May be heard in committee March 18. |
| Feb 16, 2024 | Read first time. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Aug 31, 2024 | AB 3030 Calderon Concurrence in Senate Amendments By AGUIAR-CURRY | Assembly Floor |
Passed |
|
| Aug 30, 2024 | Special Consent AB3030 Calderon By Limón | Senate Floor |
Passed |
|
| Aug 15, 2024 | Do pass as amended | Senate Standing Committee on Appropriations |
Passed |
|
| Aug 5, 2024 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Jun 26, 2024 | Do pass as amended, but first amend, and re-refer to the Committee on [Appropriations] | Senate Standing Committee on Health |
Passed |
|
| May 20, 2024 | AB 3030 Calderon Assembly Third Reading | Assembly Floor |
Passed |
|
| May 8, 2024 | Do pass. To Consent Calendar. | Assembly Standing Committee on Appropriations |
Passed |
|
| Apr 23, 2024 | Do pass as amended and be re-referred to the Committee on [Appropriations] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
|
| Apr 9, 2024 | Do pass as amended and be re-referred to the Committee on [Privacy and Consumer Protection] | Assembly Standing Committee on Health |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| ATA Action | ** Support | |
| California Nurses Association | ** Against | |
| Medical Board of California | ** Support | |
| Oakland Privacy | ** Support |
CONCURRENCE IN SENATE AMENDMENTS
CSA1 Bill Id:AB 3030
Author:(Calderon)
As Amended Ver:August 15, 2024
Majority vote
SUMMARY
Requires specified health care providers to disclose the use of a generative artificial intelligence (GenAI) tool when it is used to generate communications to a patient pertaining to patient clinical information, as defined, requires such a communication to include clear instructions permitting a patient to communicate with a human health care provider or other appropriate person, as specified, and exempts from disclosure written communications that are generated by GenAI and reviewed by a licensed or certified health care provider.
Senate Amendments
- Exempt from disclosure patient communications generated by GenAI and reviewed by a licensed or certified health care provider.
- Specify that violations by physician, health facility, and clinic are subject to enforcement via existing enforcement mechanisms of regulatory agencies including the Medical Board of California, Osteopathic Medical Board of California, and the California Department of Public Health, as appropriate.
- Define "health facility."
- Narrow the applicability of the bill to patient communications related to patient clinical information, and define "patient clinical information."
COMMENTS
- AI And GenAI. In a draft regulation, the California Privacy Protection Agency (CPPA) defines AI as follows:
AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes GenAI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
GenAI is a subset of AI that can be trained in a variety of ways and applied to a large and growing set of use cases across economic sectors. GenAI uses a type of machine learning called "deep learning" that uses multilayer neural networks, similar to the structure of a human brain, to process input and generate novel responses. LLMs are a type of GenAI model that has been specifically designed to understand, generate, and work with human language. These models are trained on vast quantities of text sourced from the internet and historical literature. GPT-4 and Copilot are examples of recently launched, publicly available interactive LLMs. Because of the extraordinary potential of GenAI applications, it will have enormous implications for industries across the economy and for labor and the workforce, as well as in daily life.
- Recent Federal and State Activity. As GenAI has exploded into popular knowledge and use, here has been a flurry of activity among states and the federal administrative and regulatory agencies. Governor Newsom issued an Executive Order (EO) in September, 2023, that addressed evaluation and deployment of AI within state government. The administration is implementing the EO, which includes procurement proposals by state agencies, two of which relate to health care. One proposal seeks to improve efficiency in inspections of health facilities by the Department of Public Health (DPH), and another within the California Health and Human Services Agency seeks to improve translations in the health care setting.
The CPPA is issued draft regulations in December 2023 that would impose requirements for businesses using "automated decision-making technology" (ADMT). However, the draft CPPA regulations on ADMT do not appear to explicitly address the disclosure issue raised by this bill.
The federal government has recently taken a number of AI-related actions. In 2023, the Federal Blueprint for an AI Bill of Rights included five principles and associated practices to protect the rights of the American public in the age of AI. One principle is that someone should know that an automated system is being used and understand how and why it contributes to outcomes that impact them. Another principle holds that a person should be able to opt out, where appropriate, and have access to someone who can quickly consider and remedy problems. This bill appears generally aligned with these principles; however, the principles also state the notice should be calibrated to the level of risk based on the context, and that automated systems should provide explanations that are technically valid, meaningful and useful.
A federal EO was issued in October 2023 to establish a government-wide effort to guide responsible AI development and deployment through federal agency leadership, regulation of industry, and engagement with international partners.
- Assembly Informational Hearing. On February 27, 2024, the Assembly Privacy and Consumer Protection Committee held an informational hearing titled, "Understanding AI: Myths, Magic, and Machine Learning." The briefing paper notes AI is already embedded into most online systems, that it is integral to many aspects of modern society, and that the advent of GenAI will undoubtedly lead to an even greater number of applications. It suggests policymakers will need to design regulatory guardrails to limit harmful uses while allowing for the development and refinement of tools that benefit society. The paper discusses negative aspects of AI that imply a role for regulation, and a variety of troubling applications of AI. Specifically, it discusses bias and discrimination, effect on labor, deepfakes, (the creation of fake but realistic text, imagery, video, and audio by GenAI), questionable originality and copyright issues, and the inability to remove data from a trained model. With respect to health care, it notes that when AI tools are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. It also notes the capacity for GenAI to "hallucinate," or generate output that has no basis in reality, is a unique risk if GenAI is embedded in health care applications. Disclosure and digital watermarking (the practice of embedding visible or invisible markers into a GenAI product) are presented as stopgap measures to prevent harms in the short term.
- GenAI in Health Care. The Government Accountability Office published a report in 2020 noting both clinical and administrative AI tools have shown promise for augmenting patient care, but can raise a number of ethical, legal, economic, and social concerns, including data access, bias, scaling and integration, lack of transparency, privacy, and uncertainty over liability.
In a January 2024 article in Health Affairs Forefront, "Generative AI In Health Care: Opportunities, Challenges, And Policy," author Niam Yaraghi notes GenAI performs optimally in environments characterized by high repetition and low risk. Routine tasks like information collecting and reporting are promising applications where AI can add significant value, while more nuanced tasks like providing treatment recommendations are unlikely in the near future.
The American Medical Association (AMA) adopted "Principles for Augmented Intelligence Development, Deployment, and Use" on November 14, 2023. With respect to transparency, AMA writes that as implementation of AI-enabled tools and systems continues to increase, it is essential that use of AI in health care be transparent to both physicians and patients. AMA notes transparency requirements should be tailored in a way that best suits the needs of the end users, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. Specifically, AMA states that when AI is used in a manner which directly impacts patient care, access to care, or medical decision making, that use of AI should be disclosed and documented to both physicians and/or patients in a culturally and linguistically appropriate manner.
According to the Author
Across the state, pilot programs are testing the use of GenAI as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our health care systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. This bill would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
Arguments in Support
The Medical Board of California (MBC) writes in support that the use of AI in the practice of medicine offers both benefits and risks to consumers. The MBC states in support that this bill promotes patient awareness of the manner and circumstances of how their care is delivered.
Arguments in Opposition
The California Nurses Association (CNA) is opposed to this bill unless substantially amended to apply to all automated communications tools, further address liability issues, change notification language, require an acknowledgement by the patient of receipt of the notice, require a health care provider to be available for direct communication within 24 hours of the receipt of a notice, and clarify it does not permit an entity to substitute the output of AI for the professional judgement of a licensed health care professional or penalize such a professional to override the AI or decline to use AI to communicate with patients. CNA argues GenAI is categorically unsafe for patient communication and warns it can be incorrect, nonsensical, or actively harmful.
FISCAL COMMENTS
According to the Senate Appropriations Committee:
- DPH estimates costs ranging from $382,000 to $881,000 in 2024-25, and ongoing costs ranging from $656,000 to $1,518,000 beginning in 2025-26 to verify compliance with generative AI patient communication disclaimers used by health facilities and clinics (Licensing and Certification Fund).
- The Department of Consumer Affairs anticipates minor and absorbable costs for the Osteopathic Medical Board of California and the MBC.
VOTES:
ASM HEALTH: 16-0-0
YES: Bonta, Waldron, Aguiar-Curry, Arambula, Wendy Carrillo, Flora, Haney, Jones-Sawyer, Maienschein, McCarty, Joe Patterson, Rodriguez, Sanchez, Santiago, Schiavo, Weber
ASM PRIVACY AND CONSUMER PROTECTION: 10-0-1
YES: Joe Patterson, Bryan, Dixon, Hoover, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson
ABS, ABST OR NV: Bauer-Kahan
ASM APPROPRIATIONS: 15-0-0
YES: Wicks, Sanchez, Arambula, Bryan, Calderon, Wendy Carrillo, Dixon, Mike Fong, Friedman, Haney, Hart, Jim Patterson, Pellerin, Ta, Villapudua
ASSEMBLY FLOOR: 71-0-9
YES: Addis, Aguiar-Curry, Alanis, Alvarez, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Davies, Dixon, Essayli, Flora, Mike Fong, Vince Fong, Friedman, Gallagher, Garcia, Gipson, Grayson, Haney, Hart, Holden, Hoover, Irwin, Jackson, Jones-Sawyer, Lackey, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Ortega, Pacheco, Papan, Jim Patterson, Joe Patterson, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Rodriguez, Blanca Rubio, Sanchez, Santiago, Schiavo, Soria, Ta, Ting, Valencia, Villapudua, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Robert Rivas
ABS, ABST OR NV: Arambula, Cervantes, Megan Dahle, Gabriel, Kalra, Mathis, Stephanie Nguyen, Luz Rivas, Zbur
UPDATED
VERSION: August 15, 2024
CONSULTANT: Lisa Murawski / HEALTH / (916) 319-2097 FN: 0004436
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | AB 3030 |
THIRD READING
Bill No: AB 3030
Author: Calderon (D), et al.
Amended: 8/15/24 in Senate
Vote: 21
SENATE HEALTH COMMITTEE: 11-0, 6/26/24
AYES: Roth, Nguyen, Glazer, Gonzalez, Grove, Hurtado, Limón, Menjivar, Rubio, Smallwood-Cuevas, Wiener
SENATE APPROPRIATIONS COMMITTEE: 7-0, 8/15/24
AYES: Caballero, Jones, Ashby, Becker, Bradford, Seyarto, Wahab
ASSEMBLY FLOOR: 71-0, 5/20/24 - See last page for vote
SUBJECT: Health care services: artificial intelligence
SOURCE: Author
DIGEST:
This bill requires a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence and clear instructions on how the patient may contact a human person.
ANALYSIS:
Existing law:
- Establishes the California Department of Public Health (CDPH) which licenses and regulates health facilities and clinics. [Health and Safety Code (HSC) §1200 et seq., §1250 et seq.]
- Defines “clinic” as an organized outpatient health facility that provides direct medical, surgical, dental, optometric, or podiatric advice, services, or treatment to patients who remain less than 24 hours, and that may also provide diagnostic or therapeutic services to patients in the home as an incident to care provided at the clinic facility. [HSC Section §1200]
- Defines “health facility” to mean a facility, place, or building that is organized, maintained, and operated for the diagnosis, care, prevention, and treatment of human illness, physical or mental, including convalescence and rehabilitation and including care during and after pregnancy, or for any one or more of these purposes, for one or more persons, to which the persons are admitted for a 24-hour stay or longer, including, but not limited to, hospitals, nursing facilities, and hospice facilities. [HSC §1250]
- Establishes the Department of Consumer Affairs, under which are various professional licensing boards within the Healing Arts that license and regulate health professionals, including the Medical Board of California and the Board of Registered Nursing, among others. [Business and Professions Code (BPC) §500 et seq., §2000 et seq., §2700 et seq.]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [CIV §1798.100 et seq.]
This bill:
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence (AI) to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include both of the following:
- A disclaimer that indicates to the patient that the communication was generated by generative AI; and,
- Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physician’s office, or office of a group provider, or other appropriate person.
- Requires the manner in which the disclaimer is provided to be as follows:
- For written communications involving physical and digital media, including letters, emails, and other occasional messages, requires the disclaimer to appear prominently at the beginning of each communication;
- For written communications involving continuous online interactions, including chat-based telehealth, requires the disclaimer to be prominently displayed throughout the interaction;
- For audio communications, requires the disclaimer to be provided verbally at the start and end of the interaction; and,
- For video communications, requires the disclaimer to be prominently displayed throughout the interaction.
- Exempts from the requirements of this bill a communication that is generated by generative AI and read and reviewed by a human licensed or certified health care provider.
- Defines the following terms for purposes of this bill:
- “Artificial intelligence” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs than can influence physical or virtual environments;
- “Generative artificial intelligence” means AI that can generate derived synthetic content, including images, videos, audio, text, and other digital content;
- “Patient clinical information” means information relating to the health status of a patient. Specifies that this information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters;
- “Clinic” has the same meaning as in the provisions of law governing clinics, and is an organized outpatient health facility that provides direct medical, surgical, dental, optometric, or podiatric advice, services, or treatment to patients who remain less than 24 hours, and that may also provide diagnostic or therapeutic services to patients in the home as an incident to care provided at the clinic facility;
- “Health facility” means a facility, place, or building that is organized, maintained, and operated for the diagnosis, care, prevention, and treatment of human illness, physical or mental, including convalescence and rehabilitation and including care during and after pregnancy, or for any one or more of these purposes, to which persons are admitted for a 24-hour stay or longer, and includes various types such as general acute care hospital, acute psychiatric hospital, and skilled nursing facilities, among others;
- “Physician’s office” means an office of a physician in solo practice;
- “Office of a group practice” means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed as a community clinic;
- “Health care provider” means a person licensed or certified under the Healing Arts.
Comments
- Author’s statement. According to the author, across the state, pilot programs are testing the use of generative AI as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our healthcare systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. This bill would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
- Information provided by author on why this bill is necessary. According to the author, the adoption of generative AI in healthcare has the potential to revolutionize patient care, diagnosis, and medical research. Nonetheless, healthcare providers who deploy generative AI have to mitigate unintended consequences, as responsible use is key to harnessing its benefits while avoiding adverse outcomes. In the Blueprint for an AI Bill of Rights, the White House Office of Science and Technology Policy identified transparency in AI as a necessary industry standard. Within transparency, the report highlights the importance of providing disclaimers to help maintain patient awareness and allow them to get the type of assistance they need through generative AI or a human. Since generative AI is a nascent sub-industry in the healthcare system, current law doesn’t regulate its use. This bill aims to send a signal to the market that having patients differentiate between generative AI and a real person via healthcare platforms is necessary. The bill also requires healthcare facilities to communicate with patients on how they can reach a real human. This is critical as healthcare is incredibly personal, and getting the right and sometimes immediate assistance is necessary. Creating this threshold is important for those developing generative AI algorithms and healthcare providers trying to contract with said developers. The ethical use of generative AI in healthcare is critical to maintaining trust and ensuring the best health outcomes from the use of generative AI.
- What is generative AI? In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
According to an article in MIT News, from the Massachusetts Institute of Technology, before the generative AI boom of the past few years, when people talked about AI, typically they were talking about machine-learning models that can learn to make a prediction based on data. For instance, such models are trained, using millions of examples, to predict whether a certain X-ray shows signs of a tumor or if a particular borrower is likely to default on a loan. Generative AI can be thought of as a machine-learning model that is trained to create new data, rather than making a prediction about a specific dataset. A generative AI system is one that learns to generate more objects that look like the data it was trained on. Generative AI systems such as ChatGPT are powerful because they are trained on extremely large datasets, which could potentially take advantage of nearly all the information on the internet. According to an article published by CalTech Science Exchange, todays generative AI models produce content that often is indistinguishable from that created by humans. Generative AI has successfully written news articles, created realistic artwork, and composed aesthetically pleasing music. While these applications sometimes make glaring mistakes (sometimes referred to as hallucinations), they are being used for many purposes, including in health care. - Generative AI in health care. According to a July 25, 2023 viewpoint article in the Journal of the American Medical Association, “Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients,” generative AI is being heralded in the medical field for its potential to ease the long-lamented burden of medical documentation by generating visit notes, treatment codes, and medical summaries. This article stated that to date, no court in the United States has considered the question of liability for medical injuries caused by relying on AI-generated information. The article warned that to minimize risks, even medical professionals must exercise caution when relying on AI-generated information, due to the “black-box” nature of how these systems generate output, because it may be impossible for the physician to independently evaluate the accuracy of the AI’s output. An article in Implement Science, published in March 2024, stated that the utility and impact of generative AI in healthcare remain poorly understood, with concerns about ethical and legal implications, integration into healthcare service delivery, and workforce utilization. The American Medical Association published “Principles for Augmented Intelligence Development, Deployment, and Use,” which was approved by the AMA Board of Trustees on November 14, 2023. According to this AMA document, as the number of AI-enabled health care tools and systems continue to grow, these technologies must be designed, developed, and deployed in a manner that is ethical, equitable, responsible, and transparent. The AMA notes that while the FDA regulates AI-enabled medical devices, many types of AI-enabled technologies fall outside the scope of FDA oversight, including AI that may have clinical applications, such as some clinical decision support functions. The AMA report specifically looked at transparency in use of AI-enabled systems, and stated that it is essential that use of AI in health care be transparent to both physicians and patients, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. The AMA report states that when AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients. The AMA noted that while transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. According to the AMA, when AI is used in a manner which directly impacts patient care, access to care, or medical decision making, that use of AI should be disclosed and documented to both physicians and patients. The opportunity for a patient to request additional review from a licensed clinician should be made available upon request. The AMA policy also states that AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee:
- CDPH estimates costs ranging from $382,000 to $881,000 in 2024-25, and ongoing costs ranging from $656,000 to $1,518,000 beginning in 2025-26 to verify compliance with generative AI patient communication disclaimers used by health facilities and clinics (Licensing and Certification Fund).
- The Department of Consumer Affairs anticipates minor and absorbable costs for the Osteopathic Medical Board of California (OMBC) and the Medical Board of California (MBC).
SUPPORT: (Verified 8/8/24)
ATA Action
Medical Board of California
Oakland Privacy
OPPOSITION: (Verified 8/8/24)
California Nurses Association
ARGUMENTS IN SUPPORT: The Medical Board of California (MBC) states in support that the use of AI in the practice of medicine offers both benefits and risks to consumers. MBC states that it appreciates that this bill promotes patient awareness of the manner and circumstances of how their care is delivered. Oakland Privacy states in support that it is a citizen’s coalition that works to defend the right to privacy, enhance public transparency, and increase oversight of law enforcement, particularly regarding the use of surveillance techniques and equipment. Oakland Privacy states that this is a simple bill that relies on common sense for its provisions. Despite the many concerns we all have with the deployment of generative AI, it seems reasonable that the software could take over many routine tasks such as check-up reminders, appointment verifications, and other practices that can benefit from automation. However, sometimes health facilities, clinics, and physicians must convey difficult and sensitive information, or respond to patients communicating symptoms, complaints, or concerns. In those cases, patients thinking they are talking with or writing to a human being when they are interacting with a generative AI is problematic. This bill addresses those gradients in the right way, by not prescribing when generative AI can and cannot be used, but mandates that if it is used that the patient must be fully informed and empowered to shift the communication to a human being if needed.
ARGUMENTS IN OPPOSITION: The California Nurses Association (CNA) is opposed unless amended, stating that disclaimers on generative AI will unintentionally legitimize the use of generative AI in health care, and urges that the legislature ensure that generative AI is never used to correspond with patients. CNA argues that it is categorically unsafe to use generative AI for patient communication and can be incorrect, nonsensical, or actively harmful. Human-to-human care and communication is essential to health care. CNA cites one study of the use of generative AI to translate discharge summaries written by, and for, clinicians into patient-friendly language, which found that the models did improve readability – but also had safety-threatening inaccuracies in 39.1% of cases, which included important omissions and “inaccurate albeit convincing outputs that are termed hallucinations.” CNA also points to commercial health care algorithms that have been found to have serious racial bias or validity problems, many of which would not have been easily identifiable by a human user or reviewer. Numerous studies have documented implicit bias in health care providers which may then be recorded in electronic health records used to train machine learning algorithms. Instead of requiring disclaimers of generative AI communications, CNA recommends that the scope of the notification be extended to all automated communications tools, or at minimum all artificial intelligence in patient communications. CNA states that human review should be used by health care entities to shield against misinformation or harm related to the use of automated communication tools. By allowing entities to notify patients that there was human review of the communication, this may provide a false sense of security to a patient if there was no meaningful review by a licensed health care professional. Without existing parameters and regulations on what constitutes meaningful human review, the notice may result in harm.
ASSEMBLY FLOOR: 71-0, 5/20/24
AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Davies, Dixon, Essayli, Flora, Mike Fong, Vince Fong, Friedman, Gallagher, Garcia, Gipson, Grayson, Haney, Hart, Holden, Hoover, Irwin, Jackson, Jones-Sawyer, Lackey, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Ortega, Pacheco, Papan, Jim Patterson, Joe Patterson, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Rodriguez, Blanca Rubio, Sanchez, Santiago, Schiavo, Soria, Ta, Ting, Valencia, Villapudua, Waldron, Wallis, Ward, Weber, Wicks, Wilson, Wood, Robert Rivas
NO VOTE RECORDED: Arambula, Cervantes, Megan Dahle, Gabriel, Kalra, Mathis, Stephanie Nguyen, Luz Rivas, Zbur
Prepared by: Vincent D. Marchand / HEALTH / (916) 651-4111
8/19/24 12:41:04
**** END ****
SENATE COMMITTEE ON APPROPRIATIONS
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
AB 3030 (Calderon) - Health care services: artificial intelligence
Version: June 27, 2024 | Policy Vote: HEALTH 11 - 0 |
Urgency: No | Mandate: No |
Hearing Date: August 15, 2024 | Consultant: Agnes Lee |
Bill Summary:
AB 3030 would impose requirements on a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence (AI) to generate written or verbal patient communications pertaining to patient clinical information.
*********** ANALYSIS ADDENDUM – SUSPENSE FILE ***********
The following information is revised to reflect amendments
adopted by the committee on August 15, 2024
Fiscal Impact:
- The California Department of Public Health (CDPH) estimates costs ranging from $382,000 to $881,000 in 2024-25, and ongoing costs ranging from $656,000 to $1,518,000 beginning in 2025-26 to verify compliance with generative AI patient communication disclaimers used by health facilities and clinics (Licensing and Certification Fund).
- The Department of Consumer Affairs anticipates minor and absorbable costs for the Osteopathic Medical Board of California (OMBC) and the Medical Board of California (MBC).
Author Amendments:
Add a co-author.
-- END --
SENATE COMMITTEE ON APPROPRIATIONS
Senator Anna Caballero, Chair
2023 - 2024 Regular Session
AB 3030 (Calderon) - Health care services: artificial intelligence
Version: June 27, 2024 | Policy Vote: HEALTH 11 - 0 |
Urgency: No | Mandate: No |
Hearing Date: August 5, 2024 | Consultant: Agnes Lee |
Bill Summary:
AB 3030 would impose requirements on a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information.
Fiscal Impact:
- Unknown costs for the California Department of Public Health (CDPH) for state administration (Licensing and Certification Fund).
- The Department of Consumer Affairs anticipates minor and absorbable costs for the Osteopathic Medical Board of California (OMBC) and the Medical Board of California (MBC).
Background:
The CDPH licenses and regulates health facilities and clinics. Existing law establishes the MBC for the licensing, regulation, and discipline of physicians and surgeons; and establishes the OMBC for the licensing and regulation of osteopathic physicians and surgeons.
Current law establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018. In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models, that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
Proposed Law:
Specific provisions of the bill would:
- Require a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include both of the following:
- A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence, as specified.
- Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physician’s office, or office of a group provider, or other appropriate person.
- Provide that if a communication is generated by generative artificial intelligence and read and reviewed by a human licensed or certified health care provider, the requirement above does not apply.
-- END --
SENATE COMMITTEE ON HEALTH
Senator Richard Roth, Chair
BILL NO: AB 3030
AUTHOR: | Calderon | ||
VERSION: | June 18, 2024 | ||
HEARING DATE: | June 26, 2024 | ||
CONSULTANT: | Vincent D. Marchand | ||
SUBJECT: Health care services: artificial intelligence
SUMMARY:
Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence and clear instructions on how the patient may contact a human person.
Existing law:
- Establishes the California Department of Public Health (CDPH) which licenses and regulates health facilities and clinics. [HSC §1200 et seq., §1250 et seq.]
- Defines “clinic” as an organized outpatient health facility that provides direct medical, surgical, dental, optometric, or podiatric advice, services, or treatment to patients who remain less than 24 hours, and that may also provide diagnostic or therapeutic services to patients in the home as an incident to care provided at the clinic facility. [HSC Section §1200]
- Defines “health facility” to mean a facility, place, or building that is organized, maintained, and operated for the diagnosis, care, prevention, and treatment of human illness, physical or mental, including convalescence and rehabilitation and including care during and after pregnancy, or for any one or more of these purposes, for one or more persons, to which the persons are admitted for a 24-hour stay or longer, including, but not limited to, hospitals, nursing facilities, and hospice facilities. [HSC §1250]
- Establishes the Department of Consumer Affairs, under which are various professional licensing boards within the Healing Arts that license and regulate health professionals, including the Medical Board of California and the Board of Registered Nursing, among others. [BPC §500 et seq., §2000 et seq., §2700 et seq.]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [CIV §1798.100 et seq.]
This bill:
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence (AI) to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include both of the following:
- A disclaimer that indicates to the patient that the communication was generated by generative AI; and,
- Clear instructions describing how a patient may contact a human health care provider, employee of the health facility, clinic, physician’s office, or office of a group provider, or other appropriate person.
- Requires the manner in which the disclaimer is provided to be as follows:
- For written communications involving physical and digital media, including letters, emails, and other occasional messages, requires the disclaimer to appear prominently at the beginning of each communication;
- For written communications involving continuous online interactions, including chat-based telehealth, requires the disclaimer to be prominently displayed throughout the interaction;
- For audio communications, requires the disclaimer to be provided verbally at the start and end of the interaction; and,
- For video communications, requires the disclaimer to be prominently displayed throughout the interaction.
- Requires the disclaimer, for a written communication that is generated by generative AI and reviewed by a human health care provider, to indicate that.
- Defines the following terms for purposes of this bill:
- “Artificial intelligence” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs than can influence physical or virtual environments;
- “Generative artificial intelligence” means AI that can generate derived synthetic content, including images, videos, audio, text, and other digital content;
- “Patient clinical information” means information relating to the health status of a patient. Specifies that this information does not include administrative matters, including, but not limited to, appointment scheduling, billing, or other clerical or business matters;
- “Clinic” has the same meaning as in the provisions of law governing clinics, and is an organized outpatient health facility that provides direct medical, surgical, dental, optometric, or podiatric advice, services, or treatment to patients who remain less than 24 hours, and that may also provide diagnostic or therapeutic services to patients in the home as an incident to care provided at the clinic facility;
- “Health facility” means a facility, place, or building that is organized, maintained, and operated for the diagnosis, care, prevention, and treatment of human illness, physical or mental, including convalescence and rehabilitation and including care during and after pregnancy, or for any one or more of these purposes, to which persons are admitted for a 24-hour stay or longer, and includes various types such as general acute care hospital, acute psychiatric hospital, and skilled nursing facilities, among others;
- “Physician’s office” means an office of a physician in solo practice;
- “Office of a group practice” means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed as a community clinic;
- “Health care provider” means a person licensed or certified under the Healing Arts.
FISCAL EFFECT:
According to the Assembly Appropriations Committee, minor and absorbable costs to CDPH to verify that facilities are complying with the requirements of this bill.
PRIOR VOTES:
Assembly Floor: | 71 - 0 |
Assembly Appropriations Committee: | 15 - 0 |
Assembly PCP Committee | 10 - 1 |
Assembly Health Committee: | 16 - 0 |
COMMENTS:
- Author’s statement. According to the author, across the state, pilot programs are testing the use of generative AI as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our healthcare systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. This bill would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
- Information provided by author on why this bill is necessary. According to the author, the adoption of generative AI in healthcare has the potential to revolutionize patient care, diagnosis, and medical research. Nonetheless, healthcare providers who deploy generative AI have to mitigate unintended consequences, as responsible use is key to harnessing its benefits while avoiding adverse outcomes. In the Blueprint for an AI Bill of Rights, the White House Office of Science and Technology Policy identified transparency in AI as a necessary industry standard. Within transparency, the report highlights the importance of providing disclaimers to help maintain patient awareness and allow them to get the type of assistance they need through generative AI or a human. Since generative AI is a nascent sub-industry in the healthcare system, current law doesn’t regulate its use. This bill aims to send a signal to the market that having patients differentiate between generative AI and a real person via healthcare platforms is necessary. The bill also requires healthcare facilities to communicate with patients on how they can reach a real human. This is critical as healthcare is incredibly personal, and getting the right and sometimes immediate assistance is necessary. Creating this threshold is important for those developing generative AI algorithms and healthcare providers trying to contract with said developers. The ethical use of generative AI in healthcare is critical to maintaining trust and ensuring the best health outcomes from the use of generative AI.
- What is generative AI? In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
According to an article in MIT News, from the Massachusetts Institute of Technology, before the generative AI boom of the past few years, when people talked about AI, typically they were talking about machine-learning models that can learn to make a prediction based on data. For instance, such models are trained, using millions of examples, to predict whether a certain X-ray shows signs of a tumor or if a particular borrower is likely to default on a loan. Generative AI can be thought of as a machine-learning model that is trained to create new data, rather than making a prediction about a specific dataset. A generative AI system is one that learns to generate more objects that look like the data it was trained on. Generative AI systems such as ChatGPT are powerful because they are trained on extremely large datasets, which could potentially take advantage of nearly all the information on the internet. According to an article published by CalTech Science Exchange, todays generative AI models produce content that often is indistinguishable from that created by humans. Generative AI has successfully written news articles, created realistic artwork, and composed aesthetically pleasing music. While these applications sometimes make glaring mistakes (sometimes referred to as hallucinations), they are being used for many purposes, including in health care. - Generative AI in health care. According to a July 25, 2023 viewpoint article in the Journal of the American Medical Association, “Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients,” generative AI is being heralded in the medical field for its potential to ease the long-lamented burden of medical documentation by generating visit notes, treatment codes, and medical summaries. This article stated that to date, no court in the United States has considered the question of liability for medical injuries caused by relying on AI-generated information. The article warned that to minimize risks, even medical professionals must exercise caution when relying on AI-generated information, due to the “black-box” nature of how these systems generate output, because it may be impossible for the physician to independently evaluate the accuracy of the AI’s output. An article in Implement Science, published in March 2024, stated that the utility and impact of generative AI in healthcare remain poorly understood, with concerns about ethical and legal implications, integration into healthcare service delivery, and workforce utilization. The American Medical Association published “Principles for Augmented Intelligence Development, Deployment, and Use,” which was approved by the AMA Board of Trustees on November 14, 2023. According to this AMA document, as the number of AI-enabled health care tools and systems continue to grow, these technologies must be designed, developed, and deployed in a manner that is ethical, equitable, responsible, and transparent. The AMA notes that while the FDA regulates AI-enabled medical devices, many types of AI-enabled technologies fall outside the scope of FDA oversight, including AI that may have clinical applications, such as some clinical decision support functions. The AMA report specifically looked at transparency in use of AI-enabled systems, and stated that it is essential that use of AI in health care be transparent to both physicians and patients, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. The AMA report states that when AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients. The AMA noted that while transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. According to the AMA, when AI is used in a manner which directly impacts patient care, access to care, or medical decision making, that use of AI should be disclosed and documented to both physicians and patients. The opportunity for a patient to request additional review from a licensed clinician should be made available upon request. The AMA policy also states that AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
- Support. The Medical Board of California (MBC) states in support that the use of AI in the practice of medicine offers both benefits and risks to consumers. MBC states that it appreciates that this bill promotes patient awareness of the manner and circumstances of how their care is delivered. Oakland Privacy states in support that it is a citizen’s coalition that works to defend the right to privacy, enhance public transparency, and increase oversight of law enforcement, particularly regarding the use of surveillance techniques and equipment. Oakland Privacy states that this is a simple bill that relies on common sense for its provisions. Despite the many concerns we all have with the deployment of generative AI, it seems reasonable that the software could take over many routine tasks such as check-up reminders, appointment verifications, and other practices that can benefit from automation. However, sometimes health facilities, clinics, and physicians must convey difficult and sensitive information, or respond to patients communicating symptoms, complaints, or concerns. In those cases, patients thinking they are talking with or writing to a human being when they are interacting with a generative AI is problematic. This bill addresses those gradients in the right way, by not prescribing when generative AI can and cannot be used, but mandates that if it is used that the patient must be fully informed and empowered to shift the communication to a human being if needed.
- Oppose unless amended. The California Radiological Society (CRS) states that AI may be utilized in many ways in the clinical setting, particularly radiology. AI can help triage imaging studies based on urgency, such as identifying critical findings like brain hemorrhage on a CT scan, and prioritize it for immediate review, thus potentially saving lives by reducing diagnostic turnaround times. AI tools can generate preliminary reports by identifying common findings in images, which are then reviewed and finalized by a radiologist, significantly speeding up the reporting process. There may be clinical notes that might be generated partially by AI but still signed off by a human, or even an email that may use AI to start a response, but is then edited and completed by the physician. Unfortunately, CRS states this bill would require providers to disclose the use of AI in any communication to the patient, even if reviewed by the provider or simply used as an aid to help the provider draft the communication. By requiring health care providers to disclose the type of tool used to assist them in servicing their patients, CRS is concerned this could potentially discourage the use of AI with the added burdens and penalties. The California Medical Association makes similar arguments, and joins CRS in requesting amendments so that the required disclosure would not apply in those instances where the communication was reviewed by a human. The California Hospital Association (CHA) shares these concerns, and additionally highlights that because this bill is adding its provisions to an area of law that also includes requirements related to requirements that hospitals and emergency physicians provide emergency care regardless of a patient’s ability to pay, the penalties for violations of this area of law are more severe than for other provisions of the law regarding health facilities, including fines of up to $25,000 per violation, which could mean $25,000 for each email or text communication that didn’t include the disclosure. In addition to the amendment to exempt communications that are reviewed by a human from the disclosure requirements, CHA is requesting amendments to make clear that the penalties in this section of law do not apply to this bill. CMA and CRS also request this amendment.
The California Nurses Association (CNA) is also opposed unless amended, but for different reasons than the above opponents. CNA is concerned that disclaimers on generative AI will unintentionally legitimize the use of generative AI in health care, and urges that the legislature ensure that generative AI is never used to correspond with patients. CNA argues that it is categorically unsafe to use generative AI for patient communication and can be incorrect, nonsensical, or actively harmful. Human-to-human care and communication is essential to health care. CNA cites one study of the use of generative AI to translate discharge summaries written by, and for, clinicians into patient-friendly language, which found that the models did improve readability – but also had safety-threatening inaccuracies in 39.1% of cases, which included important omissions and “inaccurate albeit convincing outputs that are termed hallucinations.” CNA also points to commercial health care algorithms that have been found to have serious racial bias or validity problems, many of which would not have been easily identifiable by a human user or reviewer. Numerous studies have documented implicit bias in health care providers which may then be recorded in electronic health records used to train machine learning algorithms. Instead of requiring disclaimers of generative AI communications, CNA recommends that the scope of the notification be extended to all automated communications tools, or at minimum all artificial intelligence in patient communications. CNA states that human review should be used by health care entities to shield against misinformation or harm related to the use of automated communication tools. By allowing entities to notify patients that there was human review of the communication, this may provide a false sense of security to a patient if there was no meaningful review by a licensed health care professional. Without existing parameters and regulations on what constitutes meaningful human review, the notice may result in harm. - Policy comment on enforcement provisions of Article 7. This bill adds new sections to an area of the Health and Safety Code (Article 7 of Chapter 2, Division 2) that includes the California version of the federal Emergency Medical Treatment and Active Labor Act (EMTALA). These provisions of law require emergency services and care to be provided to any person requesting the services or care, for any condition in which the person is in serious injury or illness, at any health facility that maintains and operates an emergency department. The provision of care cannot be affected by a person’s ability to pay or any other characteristic. Violations of these provisions subject hospitals to civil penalties of up to $25,000 per violation, and subjects physicians to civil penalties from the Medical Board of California of up to $5,000 per violation. While it is clear in context that these penalties are intended to enforce the state EMTALA and related provisions, under the letter of the law, because this bill would be in the same “Article 7,” these penalties could be invoked for violations of this bill. To be clear, there are also other unrelated provisions in Article 7, such as a requirement that patients be offered hepatitis-screening tests, which arguably were not intended to be subject to these penalties either. Outside of Article 7, there are general enforcement and penalty structures for licensed health facilities (such as hospitals and skilled nursing facilities), as well as an enforcement process for licensed clinics, that apply for all other requirements in the law pertaining to these licensed entities. The Committee may wish to consider amending this bill to clarify that the more severe penalties associated with Article 7 do not apply to violations of this bill.
SUPPORT AND OPPOSITION:
Support: Medical Board of California
Oakland Privacy
Oppose: California Dental Association (unless amended)
California Hospital Association (unless amended)
California Medical Association (unless amended)
California Nurses Association (unless amended)
California Radiological Society (unless amended)
-- END --
ASSEMBLY THIRD READING
ATR Bill Id:AB 3030
Author:(Calderon)
As Amended Ver:April 25, 2024
Majority vote
SUMMARY
Requires specified health care providers to disclose the use of a generative artificial intelligence (GenAI) tool when it is used to generate responses that are communicated to a patient, and requires such a communication to include clear instructions permitting a patient to communicate with a human health care provider.
Major Provisions
COMMENTS
- AI And GenAI. In a draft regulation, the California Privacy Protection Agency (CPPA) defines AI as follows:
AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes GenAI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
GenAI is a subset of AI that can be trained in a variety of ways and applied to a large and growing set of use cases across economic sectors. GenAI uses a type of machine learning called "deep learning" that uses multilayer neural networks, similar to the structure of a human brain, to process input and generate novel responses. LLMs are a type of GenAI model that has been specifically designed to understand, generate, and work with human language. These models are trained on vast quantities of text sourced from the internet and historical literature. GPT-4 and Copilot are examples of recently launched, publicly available interactive LLMs. Because of the extraordinary potential of GenAI applications, it will have enormous implications for industries across the economy and for labor and the workforce, as well as in daily life.
- Recent Federal and State Activity. As GenAI has exploded into popular knowledge and use, here has been a flurry of activity among states and the federal administrative and regulatory agencies. Governor Newsom issued an Executive Order (EO) in September, 2023, that addressed evaluation and deployment of AI within state government. The administration is implementing the EO, which includes procurement proposals by state agencies, two of which relate to health care. One proposal seeks to improve efficiency in inspections of health facilities by the Department of Public Health (DPH), and another within the California Health and Human Services Agency seeks to improve translations in the health care setting.
The CPPA is issued draft regulations in December 2023 that would impose requirements for businesses using "automated decision-making technology" (ADMT). However, the draft CPPA regulations on ADMT do not appear to explicitly address the disclosure issue raised by this bill.
The federal government has recently taken a number of AI-related actions. In 2023, the Federal Blueprint for an AI Bill of Rights included five principles and associated practices to protect the rights of the American public in the age of AI. One principle is that someone should know that an automated system is being used and understand how and why it contributes to outcomes that impact them. Another principle holds that a person should be able to opt out, where appropriate, and have access to someone who can quickly consider and remedy problems. This bill appears generally aligned with these principles; however, the principles also state the notice should be calibrated to the level of risk based on the context, and that automated systems should provide explanations that are technically valid, meaningful and useful.
A federal EO was issued in October 2023 to establish a government-wide effort to guide responsible AI development and deployment through federal agency leadership, regulation of industry, and engagement with international partners. The EO directs over 50 federal entities to engage in more than 100 specific actions to implement the guidance set forth across eight overarching policy areas, namely, safety and security, innovation and competition, worker support, consideration of AI bias and civil rights, consumer protection, privacy, federal use of AI, and international leadership. In addition, the Department of Health and Human Services through the Office of the National Coordinator for Health Information Technology published the "Health Data, Technology, and Interoperability," final rule, which includes first-of-its-kind federal requirements for AI and machine learning based predictive software in health care. The rule is focused on transparency and disclosure around the design and performance of AI systems, but does not appear to specifically address the type of disclosure proposed by this bill.
- Assembly Informational Hearing. On February 27, 2024, the Assembly Privacy and Consumer Protection Committee held an informational hearing titled, "Understanding AI: Myths, Magic, and Machine Learning." The briefing paper notes AI is already embedded into most online systems, that it is integral to many aspects of modern society, and that the advent of GenAI will undoubtedly lead to an even greater number of applications. It suggests policymakers will need to design regulatory guardrails to limit harmful uses while allowing for the development and refinement of tools that benefit society. The paper discusses negative aspects of AI that imply a role for regulation, and a variety of troubling applications of AI. Specifically, it discusses bias and discrimination, effect on labor, deepfakes, (the creation of fake but realistic text, imagery, video, and audio by GenAI), questionable originality and copyright issues, and the inability to remove data from a trained model. With respect to health care, it notes that when AI tools are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. It also notes the capacity for GenAI to "hallucinate," or generate output that has no basis in reality, is a unique risk if GenAI is embedded in health care applications. Disclosure and digital watermarking (the practice of embedding visible or invisible markers into a GenAI product) are presented as stopgap measures to prevent harms in the short term.
- GenAI in Health Care. The Government Accountability Office published a report in 2020 noting both clinical and administrative AI tools have shown promise for augmenting patient care, but can raise a number of ethical, legal, economic, and social concerns, including data access, bias, scaling and integration, lack of transparency, privacy, and uncertainty over liability.
In a January 2024 article in Health Affairs Forefront, "Generative AI In Health Care: Opportunities, Challenges, And Policy," author Niam Yaraghi notes GenAI performs optimally in environments characterized by high repetition and low risk. Routine tasks like information collecting and reporting are promising applications where AI can add significant value, while more nuanced tasks like providing treatment recommendations are unlikely in the near future.
The American Medical Association (AMA) adopted "Principles for Augmented Intelligence Development, Deployment, and Use" on November 14, 2023. With respect to transparency, AMA writes that as implementation of AI-enabled tools and systems continues to increase, it is essential that use of AI in health care be transparent to both physicians and patients. AMA notes transparency requirements should be tailored in a way that best suits the needs of the end users, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. Specifically, AMA states that when AI is used in a manner which directly impacts patient care, access to care, or medical decision making, that use of AI should be disclosed and documented to both physicians and/or patients in a culturally and linguistically appropriate manner. This bill appears aligned with AMA's principles.
According to the Author
Across the state, pilot programs are testing the use of GenAI as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our health care systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. This bill would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
Arguments in Support
There is no known support.
Arguments in Opposition
The California Nurses Association (CNA) is opposed to this bill unless substantially amended to apply to all automated communications tools, further address liability issues, change notification language, require an acknowledgement by the patient of receipt of the notice, require a health care provider to be available for direct communication within 24 hours of the receipt of a notice, and clarify it does not permit an entity to substitute the output of AI for the professional judgement of a licensed health care professional or penalize such a professional to override the AI or decline to use AI to communicate with patients. CNA argues GenAI is categorically unsafe for patient communication and warns it can be incorrect, nonsensical, or actively harmful.
FISCAL COMMENTS
According to the Assembly Appropriations Committee, minor and absorbable costs to the DPH to verify that facilities are complying with the requirements of this bill.
VOTES
ASM HEALTH: 16-0-0
YES: Bonta, Waldron, Aguiar-Curry, Arambula, Wendy Carrillo, Flora, Haney, Jones-Sawyer, Maienschein, McCarty, Joe Patterson, Rodriguez, Sanchez, Santiago, Schiavo, Weber
ASM PRIVACY AND CONSUMER PROTECTION: 10-0-1
YES: Joe Patterson, Bryan, Dixon, Hoover, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson
ABS, ABST OR NV: Bauer-Kahan
ASM APPROPRIATIONS: 15-0-0
YES: Wicks, Sanchez, Arambula, Bryan, Calderon, Wendy Carrillo, Dixon, Mike Fong, Friedman, Haney, Hart, Jim Patterson, Pellerin, Ta, Villapudua
UPDATED
VERSION: April 25, 2024
CONSULTANT: Lisa Murawski / HEALTH / (916) 319-2097 FN: 0002941
Date of Hearing: May 8, 2024
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
ABPCA Bill Id:AB 3030
Author:(Calderon) – As Amended Ver:April 25, 2024
Policy Committee: | Health | Vote: | 16 - 0 |
Privacy and Consumer Protection | 10 - 0 | ||
|
|
Urgency: No State Mandated Local Program: Yes Reimbursable: No
SUMMARY:
This bill requires health care providers, including health facilities and clinics, that use a generative artificial intelligence (GenAI) tool to generate communications with patients include in those communications a disclaimer that artificial intelligence (AI) generated the communication, and instructions on how to communicate directly with a health care provider. Failure to comply with the provisions of this bill will not result in any disciplinary actions.
FISCAL EFFECT:
Minor and absorbable costs to the California Department of Public Health to verify that facilities are complying with the requirements of this bill.
COMMENTS:
- Purpose. According to the author:
Across the state, pilot programs are testing the use of GenAI as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our healthcare systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. This bill would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
- AI and GenAI. The California Privacy Protection Agency defines AI as follows:
AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment…[G]enerative (GenAI) models, such as large language models (LLMs), can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
GenAI is a subset of AI that can be trained and applied to a large and growing set of use cases. GenAI uses a type of machine learning called “deep learning” that uses multilayer neural networks, similar to the structure of a human brain, to process input and generate novel responses.
Analysis Prepared by: Allegra Kim / APPR. / (916) 319-2081
Date of Hearing: April 23, 2024
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
ABPCA Bill Id:AB 3030 (
Author:Calderon) – As Amended Ver:April 11, 2024
AS PROPSED TO BE AMENDED
SUBJECT: Health care services: artificial intelligence
SYNOPSIS
Unlike artificial intelligence (AI), which has existed for more than 50 years, generative artificial intelligence (GenAI) is more or less a fundamentally new technology. As such, its quirks and limitations are still being actively explored. One such limitation – the capacity of GenAI systems to hallucinate, or create convincing but ultimately untrue “facts” out of whole cloth – can be amusing in certain situations and deadly serious in others. If a chatbot misspells “Berenstain Bears,” the hallucination is amusing. If a GenAI medical assistant directs an elderly patient to open their medicine cabinet and swallow thirty tablets of Tylenol, the hallucination becomes deadly serious. The latter situation has not yet occurred, but combining the known limitations of GenAI technology with its rapid adoption across industries will inevitably have unforeseen consequences.
This bill seeks to combat such high-risk scenarios. It would require health care providers to disclose their use of GenAI when communicating with patients, so that patients are able to apply an appropriate degree of skepticism to any instructions provided or information requested. Committee amendments would expand the types of communications affected to include all written or verbal communications, and would specify how disclosure should be provided in a variety of common communication scenarios. Committee amendments also adjust the definition of “GenAI,” insert definitions for “AI” and “health care provider,” and remove liability exemptions from the language of the bill.
This bill is author-sponsored and has no support or opposition.
SUMMARY: Requires health practitioners to disclose their use of GenAI to generate patient communications. Specifically, this bill:
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses GenAI to generate patient communications through an online interface, or via telephone, to ensure those communications include:
- A disclaimer that indicates the communication was generated by AI. If the communication was generated by GenAI and reviewed by a human health care provider, the disclaimer may indicate this.
- Clear instructions enabling a patient to navigate to the health entity’s internet website, or other platform, in order to communicate directly with a health care provider.
- Prevents an entity or health care provider that fails to comply with the requirements of the bill from being subjected to any disciplinary actions related to licensure or certification solely because of their failure to comply.
- States that existing Health and Safety Code Section 1290, which punishes as an infraction or misdemeanors specified violations, does not apply to a violation of this bill’s provisions.
- Defines “generative artificial intelligence tool” to mean a tool that uses machine learning systems capable of generating text, images, code or other types of content, often in response to a prompt entered by a user.
EXISTING LAW:
- Establishes the California Department of Public Health which, among other functions, licenses and regulates health facilities. Defines “health facility” to mean a facility, place, or building that is organized, maintained, and operated for the diagnosis, care, prevention, and treatment of human illness, physical or mental, including convalescence and rehabilitation and including care during and after pregnancy, or for any one or more of these purposes, for one or more persons, to which the persons are admitted for a 24-hour stay or longer, including, but not limited to, hospitals, nursing facilities, and hospice facilities. (Health & Saf. Code § 1250.)
- Establishes the Department of Consumer Affairs, which licenses and oversees health care professionals through various healing arts boards. (Bus. & Prof. Code § 100, Bus. & Prof. Code § 500 et seq.)
- Outlines penalties resulting from various violations of the Health and Safety Code. (Health & Saf. Code § 1290.)
- Defines “clinic” to mean an organized outpatient health facility that provides direct medical, surgical, dental, optometric, or podiatric advice, services, or treatment to patients who remain less than 24 hours, and that may also provide diagnostic or therapeutic services to patients in the home as an incident to care provided at the clinic facility. (Health & Saf. Code § 1200.)
FISCAL EFFECT: As currently in print, this bill is keyed fiscal.
COMMENTS:
1) “But doctor, I am Pagliacci!” The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI in particular can generate compelling text, videos, and audio in an instant – but with novel technologies come novel safety concerns. As GenAI-enabled products and systems become embedded into California’s most private, sensitive spaces – healthcare facilities, for example – the Legislature may wish to carefully consider the various benefits and risks associated with these technologies. There are three primary risks related to the use of AI in healthcare:
- Bias: broadly affects AI, including GenAI.
- Memory: broadly affects AI, especially GenAI.
- Hallucination: predominantly affects GenAI.
- What is GenAI? AI refers to the mimicking of human intelligence by artificial systems, such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet. A hypothetical product that a healthcare provider might use to generate patient communications would be an example of GenAI.
Large language models. Large language models (LLMs) are a type of GenAI that have been specifically designed to understand, generate, and work with human language. These models are trained on vast quantities of text sourced from the internet and historical literature, including medical textbooks and other healthcare-related information. Chatbots that produce clear, coherent text or speech in response to user queries are considered to be LLMs.
- Risk 1: Bias. There is a common saying in computer science: “garbage in, garbage out.” The performance of an AI tool is directly impacted by the quality, quantity, and relevance of the data used to train it. Deployed with care, AI should be able to reduce the bias introduced by humans into decision-making processes; if deployed thoughtlessly or maliciously, however, these same tools have the potential to amplify historical inequities and harm vulnerable populations.
AI and healthcare. When AI tools are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. In 2007, an algorithm was developed to help doctors estimate whether it was safe for people who had delivered previous children through cesarean section to deliver subsequent children vaginally – a risky procedure. The algorithm considered various health relevant factors as it made its recommendation, such as the woman’s age, her reason for the previous cesarean, and how long ago the cesarean had been performed. However, a 2017 study found that the original algorithm was biased; it predicted Black and Latino people were less likely to have a successful vaginal birth after a cesarean than non-Hispanic white women. As a result, doctors performed more cesareans on Black and Latino people than on white people.[1] Such discrepancies tend to perpetuate historical biases – Black Americans, for example, have historically received a lower standard of healthcare than their white counterparts.[2]
Voice recognition systems. Beyond health outcomes, the very interactions between humans and GenAI can themselves be subject to bias. If a voice-recognition system is trained using audio data that excludes certain accents, it can fail to understand accented speech once deployed:
“Clow-dia,” I say once. Twice. A third time. Defeated, I say the Americanized version of my name: “Claw-dee-ah.” Finally, Siri recognizes it. Having to adapt our way of speaking to interact with speech recognition technologies is a familiar experience for people whose first language is not English or who do not have conventionally American-sounding names. I have even stopped using Siri because of it.[3]
A 2020 University of Michigan study examined the ability of several common automated speech recognition (ASR) systems to understand accents. It tested five state-of-the-art ASR systems – developed by Amazon, Apple, Google, IBM, and Microsoft – to transcribe structured interviews with 42 white speakers and 73 Black speakers. The study found that all five ASR systems exhibited substantial racial disparities.[4]
- Risk 2: Memory. Just as humans cannot intentionally forget information they have learned, it is not currently possible to remove data from trained AI.[5] Unlike an Excel spreadsheet, which stores data in neat columns, AI stores data in unpredictable and irreversible ways. The only foolproof way to remove data from an AI is to retrain it from scratch.
For a healthcare-related GenAI system to be useful, it will likely need to access patient information. Medical information is some of the most private, sensitive information to exist; coupled with a tool that possesses an elephantine memory, the widespread adoption of GenAI in healthcare threatens to be a privacy nightmare.
- Risk 3: Hallucination. When an AI produces a result that is not grounded in reality, it is said to be “hallucinating.” Text-generators such as ChatGPT do not fundamentally understand the text they are producing. They calculate one word or symbol at a time – if they estimate that the next word/symbol to output should be a period, then the sentence ends. Otherwise, the sentence continues. It is a testament to the ingenious architecture of these systems that their outputs are remotely coherent. But while the text these systems produce is cogent, it is not always correct.
“These systems live in a world of language,” said Melanie Mitchell, an A.I. researcher at the Santa Fe Institute. “That world gives them some clues about what is true and what is not true, but the language they learn from is not grounded in reality. They do not necessarily know if what they are generating is true or false.”[6]
This capacity to hallucinate is seriously concerning when an AI tool is embedded in a healthcare system. Doctor-patient trust is a critical component of the medical process; patients need to know that the precise, complicated sets of instructions they are often given by their doctors can be believed. Patients should know when a trusted healthcare provider is on the other end of a communication, and when a communication has instead been produced by GenAI.
Can a GenAI communication tool explain its reasoning? Exactly how GenAI tools arrive at conclusions and generate outputs is not always obvious. These systems are sometimes criticized for being “black boxes” whose internal logic cannot be clearly explained. If a decision-making process cannot be clearly understood, can the decision reached be fully trusted? Experts are currently researching ways to solve this problem. In the meantime, policymakers should carefully consider how and when to limit the use of “black box” AI.
- Author’s statement. According to the author:
Across the state, pilot programs are testing the use of generative Artificial Intelligence (GenAI) as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our healthcare systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. AB 3030 would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
- What this bill would do. This bill would require health care providers to disclose their use of GenAI when communicating with patients through online interfaces or over the phone, so that patients are able to apply an appropriate degree of skepticism to the instructions provided or information requested. Healthcare providers would be exempted from liability related to licensure solely on the basis of failing to disclose the use of GenAI in a patient communication.
- Analysis. This bill adopts an efficient approach to regulating GenAI: it identifies a high-risk use case for the technology, and it outlines how the technology’s use should be adjusted in order to minimize risk. In collaboration with the author, this committee has proposed a series of amendments aimed at clarifying and strengthening the provisions of this bill.
Communication types. This bill addresses two types of communications between healthcare providers and patients: communications through online interfaces and telephone calls. These two examples are representative of broader categories, namely verbal and written communications. A proposed amendment would expand the types of communication this bill affects, as well as describe exactly how disclaimers should be provided in several common communication situations: occasional physical or digital communications (mail, email, messages in patient portals), continuous digital communications (chatbots), audio communications (telephone calls and voice messages), and visual communications (video telehealth).
Liability exemptions. As written, this bill exempts healthcare providers from consequences related to licensure or certification solely based on a failure to provide appropriate disclaimers. The bill also broadly exempts healthcare providers from penalties outlined in Health and Safety Code section 1290. It is not clear why these exemptions should exist for GenAI-related communication tools specifically. As outlined above, communications provided by GenAI systems should, in fact, be treated with more caution and care than communications written or dictated by human healthcare providers. Patients should be aware of the fact that a healthcare provider was not directly involved in crafting the directions they receive, or in requesting their personal medical information. Such exemptions, if enacted into law, would likely incentivize the adoption of GenAI technology at the cost of patient safety. Furthermore, these exemptions arguably undermine the bill’s goal of ensuring patients are aware when GenAI, rather than a medical professional, is the source of the information or instructions they are receiving. A proposed amendment would remove this exemption from liability.
Defining GenAI. Few serious attempts have been made to define GenAI for legislative purposes. The Biden White House’s AI executive order defines it thus:
“Generative AI” means the class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital content.
For the purposes of this bill, this Committee proposes a related definition:
“Generative artificial intelligence” means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
This definition declines to mention that GenAI “emulates the structure and characteristics of input data.” It is true that GenAI emulates the structure and characteristics of training data – but with respect to input data, as the term is usually used, GenAI is able to produce fundamentally different outputs. For example, a GenAI system may accept text data as an input and produce visual data as an output.
Other definitions. Committee amendments propose to add definitions for “artificial intelligence” and “health care provider” into the bill. Various other minor clarifying changes have also been made, as reflected below.
- Proposed amendments. The author has agreed to amendments, discussed above, that would revise and recast the bill in its entirety. The full draft text of the amended bill is reproduced below:
SECTION 1. Section 1316.9 is added to the Health and Safety Code, to read:
1316.9. (a) A health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications shall ensure that those communications include both of the following:
(1) A disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence.
(i) For written communications involving physical and digital media, such as letters, emails, and other occasional messages, the disclaimer shall appear prominently at the start of each communication.
(ii) For written communications involving continuous online interactions, such as chat-based telehealth, the disclaimer shall be prominently displayed throughout the interaction.
(iii) For audio communications, the disclaimer shall be provided verbally at the start and end of the interaction.
(iv) For video communications, the disclaimer shall be prominently displayed throughout the interaction.
(2) Clear instructions permitting a patient to communicate with a human health care provider.
(b) If a written communication is generated by generative artificial intelligence and reviewed by a human health care provider, the disclaimer shall indicate that the communication was generated by generative artificial intelligence and reviewed by a human.
(c) For purposes of this section, the following definitions apply:
(1) “Clinic” has the same meaning as defined in Section 1200.
(2) “Artificial intelligence” means an engineered or machine based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(3) “Generative artificial intelligence” means artificial intelligence that can generate derived synthetic content, including images, videos, audio, text, and other digital content.
(4) “Office of a group practice” means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(5) “Physician’s office” means an office of a physician in solo practice.
(6) “Health care provider” means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
- Related legislation. AB 2811 (Lowenthal, 2024) would require attorneys to disclosure their use of GenAI in the composition of court documents. This bill is pending in Assembly Judiciary Committee.
AB 2930 (Bauer-Kahan, 2024) would require deployers of AI-powered automated decision tools to disclose the use of these tools when making a consequential decision. This bill is pending in Assembly Judiciary Committee.
AB 3211 (Wicks, 2024) would require conversational AI systems to disclose their creation of synthetic content to users. This bill is pending in Assembly Judiciary Committee.
SB 942 (Becker, 2024) would require covered providers of generative AI systems to include visible disclosures in their outputs. This bill is pending in Senate Governmental Organization Committee.
SB 1154 (Hurtado, 2024) would require a person that uses a pricing algorithm to recommend or set a price or commercial term to clearly disclose that fact to the customer, current or prospective employee, or independent contractor. This bill is pending in Senate Judiciary Committee.
SB 1229 (Nguyen, 2024) would require a property and casualty insurer to disclose to an applicant or insured entity if it uses AI to make decisions related to applications for residential property insurance, automobile insurance, claims review, or other coverage of losses. This bill contains an urgency clause and is pending in Senate Insurance Committee.
REGISTERED SUPPORT / OPPOSITION:
Support
None on file.
Opposition
None on file.
Analysis Prepared by: Slater Sharp / P. & C.P. / (916) 319-2200
Caleb J. Colón-Rodríguez, “Shedding Light on Healthcare Algorithmic and Artificial Intelligence Bias,” U.S. Department of Health & Human Services Office of Minority Health, Jul. 12, 2023, minorityhealth.hhs.gov/news/shedding-light-healthcare-algorithmic-and-artificial-intelligence-bias. ↑
California Task Force to Study and Develop Reparation Proposals for African American, “Final Report,” California. ↑
Claudia Lopez Lloreda, “Speech Recognition Tech Is Yet Another Example of Bias,” Jul. 5, 2020, www.scientificamerican.com/article/speech-recognition-tech-is-yet-another-example-of-bias/. ↑
Allison Koenecke et al., “Racial disparities in automated speech recognition,” PNAS, vol. 117, no. 14, p. 7684-7689, Mar. 23, 2020. ↑
Stephen Pastis, “A.I.’s un-learning problem: Researchers say it’s virtually impossible to make an A.I. model ‘forget’ the things it learns from private user data,” Yahoo! Finance, Aug. 30, 2023, finance.yahoo.com/news/un-learning-problem-researchers-virtually-164342971.html. ↑
Cade Metz, “What Makes A.I. Chatbots Go Wrong?,” New York Times, Mar. 29, 2023, www.nytimes.com/2023/03/29/technology/ai-chatbots-hallucinations.html ↑
Date of Hearing: April 9, 2024
ASSEMBLY COMMITTEE ON HEALTH
Mia Bonta, Chair
ABPCA Bill Id:AB 3030
Author:((Calderon) – As Amended Ver:March 21, 2024
SUBJECT: Health care services: artificial intelligence.
SUMMARY: Requires specified health care providers to disclose the use of a generative artificial intelligence (GenAI) tool when it is used to generate responses that are communicated to a patient. Specifically, this bill:
- Requires a health facility, clinic, physician’s office, or office of a group practice (provider) that uses a GenAI tool to generate online or telephone responses for health care providers to communicate with patients to include both of the following in such communication:
- A disclaimer that indicates to the patient that a communication was generated by artificial intelligence; and,
- Clear instructions for the patient to navigate the entity’s internet website or other platforms to communicate directly with a health care provider (defined as communication without using responses generated by artificial intelligence).
- Defines facilities subject to the requirements in 1) above, by applying a current-law definition of health facility (i.e., a facility to which patients are admitted for a 24-hour stay or longer, including hospitals, nursing facilities, and hospice facilities).
- Allows, if a response communicated through an online interface is reviewed by a human health care provider, the applicable provider to indicate that the response was generated by artificial intelligence and reviewed by a human.
- Exempts providers who do not comply with the above requirements from disciplinary action related to licensure or certification, as well as from civil and criminal liability, related to their noncompliance.
- Defines “GenAI tool” as a tool that uses machine learning systems capable of generating text, images, code, or other types of content, often in response to a prompt entered by a user.
EXISTING LAW:
- Establishes the California Department of Public Health (DPH) which, among other functions, licenses and regulates health facilities. Defines “health facility” to mean a facility, place, or building that is organized, maintained, and operated for the diagnosis, care, prevention, and treatment of human illness, physical or mental, including convalescence and rehabilitation and including care during and after pregnancy, or for any one or more of these purposes, for one or more persons, to which the persons are admitted for a 24-hour stay or longer, including, but not limited to, hospitals, nursing facilities, and hospice facilities. [Health and Safety Code § 1250]
- Establishes the Department of Consumer Affairs, which licenses and oversees health care professionals through various healing arts boards [Business and Professions Code (BPC) § 100 and BPC§ 500 et seq.]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [Civil Code § 1798.100 et seq.]
FISCAL EFFECT: None.
COMMENTS:
- PURPOSE OF THIS BILL. According to the author, across the state, pilot programs are testing the use of GenAI as a tool to assist clinicians with patient communications. As AI becomes increasingly integrated in our healthcare systems, it is important to maintain the trust between a patient and their provider, while ensuring the accuracy of information being communicated to patients. This bill would require healthcare providers who use this technology to provide a disclaimer that the communication was AI-generated, along with clear instructions for how a patient can directly communicate with a healthcare provider.
- BACKGROUND.
- AI And GenAI. In a draft regulation, the CPPA defines AI as follows:
AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative (GenAI) models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
GenAI is a subset of AI that can be trained in a variety of ways and applied to a large and growing set of use cases in across economic sectors. GenAI uses a type of machine learning called “deep learning” that uses multilayer neural networks, similar to the structure of a human brain, to process input and generate novel responses. LLMs are a type of GenAI model that has been specifically designed to understand, generate, and work with human language. These models are trained on vast quantities of text sourced from the internet and historical literature. GPT-4 and Copilot are examples of recently launched, publicly available interactive LLMs. Because of the extraordinary potential of novel GenAI applications, it will have enormous implications for industries across the economy and for labor and the workforce, as well as in daily life.
- Recent Federal and State Activity. As GenAI has exploded into popular knowledge and use, here has been a flurry of activity among states and the federal administrative and regulatory agencies. Below is a high-level overview of some recent activity:
- State Executive Order (EO). In September 2023, Governor Gavin Newsom signed an EO to study the development, use, and risks of AI technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government. The EO includes a number of provisions:
- Risk-Analysis Report: Direct state agencies and departments to perform a joint risk-analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- Procurement Blueprint: To support a safe, ethical, and responsible innovation ecosystem inside state government, agencies will issue general guidelines for public sector procurement, uses, and required training for application of GenAI – building on the White House’s Blueprint for an AI Bill of Rights and the National Institute for Science and Technology’s AI Risk Management Framework. State agencies and departments will consider procurement and enterprise use opportunities where GenAI can improve the efficiency, effectiveness, accessibility, and equity of government operations.
- Beneficial Uses of GenAI Report: Direct state agencies and departments to develop a report examining the most significant and beneficial uses of GenAI in the state. The report will also explain the potential harms and risks for communities, government, and state government workers.
- Deployment and Analysis Framework: Develop guidelines for agencies and departments to analyze the impact that adopting GenAI tools may have on vulnerable communities. The state will establish the infrastructure needed to conduct pilots of GenAI projects, including California Department of Technology approved environments or “sandboxes” to test such projects.
- State Employee Training: To support California’s state government workforce and prepare for the next generation of skills needed to thrive in the GenAI economy, agencies will provide trainings for state government workers to use state-approved GenAI to achieve equitable outcomes, and will establish criteria to evaluate the impact of GenAI to the state government workforce.
- GenAI Partnership and Symposium: Establish a formal partnership with the University of California, Berkeley and Stanford University to consider and evaluate the impacts of GenAI on California and what efforts the state should undertake to advance its leadership in this industry. The state and the institutions will develop and host a joint summit in 2024 to engage in meaningful discussions about the impacts of GenAI on California and its workforce.
- Legislative Engagement: Engage with Legislative partners and key stakeholders in a formal process to develop policy recommendations for responsible use of AI, including any guidelines, criteria, reports, and/or training.
- Evaluate Impacts of AI on an Ongoing Basis: Periodically evaluate for potential impact of GenAI on regulatory issues under the respective agency, department, or board’s authority and recommend necessary updates as a result of this evolving technology.
- Risk-Analysis Report: Direct state agencies and departments to perform a joint risk-analysis of potential threats to and vulnerabilities of California’s critical energy infrastructure by the use of GenAI.
- State Executive Order (EO). In September 2023, Governor Gavin Newsom signed an EO to study the development, use, and risks of AI technology throughout the state and to develop a deliberate and responsible process for evaluation and deployment of AI within state government. The EO includes a number of provisions:
- Recent Federal and State Activity. As GenAI has exploded into popular knowledge and use, here has been a flurry of activity among states and the federal administrative and regulatory agencies. Below is a high-level overview of some recent activity:
The administration is implementing the EO, including moving forward to evaluate procurement proposals by state agencies, two of which relate to health care: one proposal to improve efficiency in inspections of health facilities by DPH, and another within the California Health and Human Services Agency to improve translations.
- CPPA Pending Regulations. The CPPA is charged with protecting the privacy of Californians pursuant to landmark privacy-related ballot measures passed in 2018 and 2020. CPPA proposed draft regulations in December 2023, which would impose requirements for businesses using “automated decision-making technology” (ADMT) in any of the following ways:
- For decisions that tend to have the most significant impacts on consumers' lives. This would include, for example, decisions about their employment or compensation.
- Profiling an employee, contractor, applicant, or student. This would include, for example, using a keystroke logger to analyze their performance, and tracking their location.
- Profiling consumers in publicly accessible places, such as shopping malls, medical offices, and stadiums. This would include, for example, using facial-recognition technology or automated emotion assessment to analyze consumers’ behavior.
- Profiling a consumer for behavioral advertising. This would include, for example, evaluating consumers’ personal preferences and interests to display advertisements to them.
- CPPA Pending Regulations. The CPPA is charged with protecting the privacy of Californians pursuant to landmark privacy-related ballot measures passed in 2018 and 2020. CPPA proposed draft regulations in December 2023, which would impose requirements for businesses using “automated decision-making technology” (ADMT) in any of the following ways:
For the above uses of ADMT, the draft regulations would provide consumers with specified protections. However, the draft CPPA regulations on ADMT do not appear to explicitly address the disclosure issue raised by this bill.
- Federal Blueprint for an AI Bill of Rights. According to the White House, the 2023 Blueprint for an AI Bill of Rights is a set of five principles and associated practices to help guide the design, use, and deployment of automated systems to protect the rights of the American public in the age of artificial intelligence. Developed through extensive consultation with the American public, the principles are a blueprint for building and deploying automated systems that are aligned with democratic values and protect civil rights, civil liberties, and privacy. The five principles are:
- Safe and Effective Systems. You should be protected from unsafe or ineffective systems.
- Algorithmic Discrimination Protections. You should not face discrimination by algorithms and systems should be used and designed in an equitable way.
- Data Privacy. You should be protected from abusive data practices via built-in protections and you should have agency over how data about you is used.
- Notice and Explanation. You should know that an automated system is being used and understand how and why it contributes to outcomes that impact you.
- Human Alternatives, Consideration, and Fallback. You should be able to opt out, where appropriate, and have access to a person who can quickly consider and remedy problems you encounter.
- Federal Blueprint for an AI Bill of Rights. According to the White House, the 2023 Blueprint for an AI Bill of Rights is a set of five principles and associated practices to help guide the design, use, and deployment of automated systems to protect the rights of the American public in the age of artificial intelligence. Developed through extensive consultation with the American public, the principles are a blueprint for building and deploying automated systems that are aligned with democratic values and protect civil rights, civil liberties, and privacy. The five principles are:
This bill appears generally aligned with principles (4) and (5) above described in the federal Blueprint for an AI Bill of Rights, in that it requires notification that an AI was used, and requires access to a non-AI fallback option. For instance, principle (4) above indicates “you should know how and why an outcome impacting you was determined by an automated system, including when the automated system is not the sole input determining the outcome.” However, to some extent the “devil is in the details,” as principle (4) above also states the notice should be calibrated to the level of risk based on the context, and notes automated systems should provide explanations that are technically valid, meaningful and useful. Given limited attention, the possibility of notification fatigue, and a range of reactions someone might have to such a disclosure, it certainly may be the case, but it is not self-evident, that the disclosure proposed by this bill meets the criteria of being useful and meaningful to the end user.
- Federal EO. As summarized by the Congressional Research Service, on October 30, 2023, the Biden Administration released EO 14110 on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The EO establishes a government-wide effort to guide responsible AI development and deployment through federal agency leadership, regulation of industry, and engagement with international partners. The EO directs over 50 federal entities to engage in more than 100 specific actions to implement the guidance set forth across eight overarching policy areas:
- Safety and security. The EO promotes the development and implementation of repeatable processes and mechanisms to understand and mitigate risks related to AI adoption, including with respect to biosecurity, cybersecurity, national security, and critical infrastructure.
- Innovation and competition. The EO compels actions to attract AI talent to the United States, understand novel intellectual property questions, protect inventors and creators, and promote AI innovation, including at startups and small businesses.
- Worker support. The EO states that AI adoption may be disruptive to the workforce and directs agencies to research and develop potential mitigations against such disruptions.
- Consideration of AI bias and civil rights. The EO states that AI models may perpetuate biases and their implementation may lead to civil rights violations. The EO includes a section on equity and civil rights considerations for use of AI in the criminal justice system and the administration of federal government programs and benefits.
- Consumer protection. The EO instructs agencies to enforce existing, technology-agnostic authorities in an effort to minimize harms to consumers, and to identify needed authorities related to AI.
- Privacy. The EO calls for the evaluation and mitigation of privacy risks—potentially exacerbated by AI— associated with the collection, use, and retention of user data.
- Federal use of AI. The EO requires the Office of Management and Budget to establish an interagency council to coordinate AI use by federal agencies and develop guidance on AI governance and risk management activities for agencies. It acknowledges the ubiquity of GenAI tools, and directs agencies to move toward adoption with safeguards in place. The EO also calls for additional agency hiring and training activities to increase the AI workforce capacity across the federal government.
- International leadership. The EO declares that the United States should be a global leader in AI development and adoption by engaging with international allies and partners, leading efforts to develop common AI regulatory and accountability principles, and advancing responsible global technical standards for AI.
- Federal Health Data, Technology, and Interoperability Final Rule. On January 9, 2024, the Department of Health and Human Services through the Office of the National Coordinator for Health Information Technology (ONC) published the “Health Data, Technology, and Interoperability,” final rule, which includes first-of-its-kind federal requirements for AI and machine learning-(ML) based predictive software in health care. ONC states it believes AI and ML—increasingly powered by data from certified health IT and injected into day-to-day workflows within certified health IT—will have a growing impact on how health decisions are made, particularly those directly affecting patients’ lives. ONC notes the technology should be deployed in a manner that is fair, appropriate, valid, effective, and safe (“FAVES”). They state the final rule based on a rapidly growing industry consensus that greater transparency into how AI is developed, evaluated, and performs is a crucial first step to achieving responsible and safe use in health care. ONC defines a predictive decision support intervention or “Predictive DSI” as a technology that supports decision-making based on algorithms or models that derive relationships from training data and then produce an output that results in prediction, classification, recommendation, evaluation, or analysis. The rule states users of certified health IT must be able to access complete and up-to-date technical and performance information—referred to as source attributes—for predictive DSIs made available to them. It also requires risk management practices be adopted, and specifies that risk management practices must cover topics including validity, reliability, robustness, fairness, intelligibility, safety, security, and privacy. ONC concludes that wide adoption of responsible AI in health care is a shared responsibility across numerous stakeholders.
- Federal EO. As summarized by the Congressional Research Service, on October 30, 2023, the Biden Administration released EO 14110 on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. The EO establishes a government-wide effort to guide responsible AI development and deployment through federal agency leadership, regulation of industry, and engagement with international partners. The EO directs over 50 federal entities to engage in more than 100 specific actions to implement the guidance set forth across eight overarching policy areas:
The rule is focused on transparency and disclosure around the design and performance of AI systems, but does not appear to specifically address the type of disclosure proposed by this bill.
- Assembly Informational Hearing. On February 27, 2024, the Assembly Privacy and Consumer Protection Committee held an informational hearing titled, “Understanding AI: Myths, Magic, and Machine Learning.” The briefing paper notes AI is already embedded into most online systems, that it is integral to many aspects of modern society, and that the advent of GenAI will undoubtedly lead to an even greater number of applications. It also notes, however, that AI is not an inherently benevolent technology – it is a tool, and it can be used for good or ill. It suggests policymakers will need to design regulatory guardrails to limit harmful uses while allowing for the development and refinement of tools that benefit society. The paper discusses negative aspects of AI that imply a role for regulation, and a variety of troubling applications of AI. Specifically, it discusses bias and discrimination, effect on labor, deepfakes, (the creation of realistic text, imagery, video, and audio by GenAI), questionable originality and copyright issues, and the inability to remove data from a trained model. With respect to health care, it notes that when AI tools are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. It also notes the capacity for GenAI to “hallucinate,” or generate output that has no basis in reality, is a unique risk if GenAI is embedded in health care applications. Disclosure and digital watermarking (the practice of embedding visible or invisible markers into a GenAI product) are presented as stopgap measures to prevent harms in the short term.
- GenAI in Health Care. The Government Accountability Office (GAO) published a report in 2020 noting AI tools have shown promise for augmenting patient care in the following two areas:
- Clinical AI tools have shown promise in predicting health trajectories of patients, recommending treatments, guiding surgical care, monitoring patients, and supporting population health management (i.e., efforts to improve the health outcomes of a community). These tools are at varying stages of maturity and adoption, but many, with the exception of population health management tools, had not achieved widespread use at that time.
- Administrative AI tools have shown promise in reducing provider burden and increasing efficiency by recording digital notes, optimizing operational processes, and automating laborious tasks. These tools are also at varying stages of maturity and adoption, ranging from emerging to widespread.
However, GAO also noted use of AI-enabled tools in health care raises a variety of ethical, legal, economic, and social concerns. GAO identified the following challenges surrounding AI tools, which may impede their widespread adoption:
- Data access. Developers experience difficulties obtaining the high-quality data needed to create effective AI tools.
- Bias. Limitations and bias in data used to develop AI tools can reduce their safety and effectiveness for different groups of patients, leading to treatment disparities.
- Scaling and integration. AI tools can be challenging to scale up and integrate into new settings because of differences among institutions and patient populations.
- Lack of transparency. AI tools sometimes lack transparency—in part because of the inherent difficulty of determining how some of them work, but also because of more controllable factors, such as the paucity of evaluations in clinical settings.
- Privacy. As more AI systems are developed, large quantities of data will be in the hands of more people and organizations, adding to privacy risks and concerns.
- Uncertainty over liability. The multiplicity of parties involved in developing, deploying, and using AI tools is one of several factors that have rendered liability associated with the use of AI tools uncertain. This may slow adoption and impede innovation.
To address the challenges, GAO identified policy options including:
- Best Practices (Policymakers could encourage relevant stakeholders and experts to establish best practices for development, implementation, and use of AI technologies); and,
- Oversight Clarity (Policymakers could collaborate with relevant stakeholders to clarify appropriate oversight mechanisms).
In a January 2024 article in Health Affairs Forefront, “Generative AI In Health Care: Opportunities, Challenges, And Policy,” author Niam Yaraghi notes GenAI performs optimally in environments characterized by high repetition and low risk. This effectiveness stems from the technology's reliance on historical data to identify patterns and make predictions, under the premise that future conditions will mirror those of the past. Using that framework, the article assesses the following applications for suitability in the health care setting:
- Information Collection and Reporting. GenAI can perform suitably for routine information gathering, such as collecting the medical histories of their patients by posing specific questions in a conversational manner.
- Diagnostics. Although there are promising applications in health care diagnostics, providers should exercise caution in deploying generative AI for diagnostics until they can train the AI on extensive medical (non-public) datasets.
- Treatment. AI currently lacks the advanced technological capability to replicate the nuanced tasks physicians perform. Treatments are often highly individualized, which does not align with AI's strengths in high-repetition, low-risk tasks. Given these complexities and risk, the integration of AI into medical treatment processes appears unlikely in the near future.
- Post-Treatment Monitoring and Follow-Up. This is a promising area that can encourage better two-way communication and better adherence to a treatment plan.
- Population Health Management. AI has significant potential in population health management; however, the use of predictive analytics must be employed with care and may be enhanced with integration of information gathered from wearable technologies and smart devices.
The article offers policy recommendations, including emphasizing transparency, informed consent, and the exchange of data through Health Information Exchange to enhance deployment of AI systems capable of learning from vast and diverse medical records.
- American Medical Association (AMA) Principles. The AMA Board of Trustees adopted “Principles for Augmented Intelligence Development, Deployment, and Use” on November 14, 2023. With respect to transparency, AMA writes that as implementation of AI-enabled tools and systems continues to increase, it is essential that use of AI in health care be transparent to both physicians and patients. Transparency requirements should be tailored in a way that best suits the needs of the end users. Disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. When AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients, and to allow each to understand how decisions impacting patient care or access to care are made. While transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. In addition, AMA writes:
- When AI is used in a manner which directly impacts patient care, access to care, or medical decision making, that use of AI should be disclosed and documented to both physicians and/or patients in a culturally and linguistically appropriate manner. The opportunity for a patient or their caregiver to request additional review from a licensed clinician should be made available upon request.
- When AI is used in a manner which directly impacts patient care, access to care, medical decision making, or the medical record, that use of AI should be documented in the medical record.
- AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
- American Medical Association (AMA) Principles. The AMA Board of Trustees adopted “Principles for Augmented Intelligence Development, Deployment, and Use” on November 14, 2023. With respect to transparency, AMA writes that as implementation of AI-enabled tools and systems continues to increase, it is essential that use of AI in health care be transparent to both physicians and patients. Transparency requirements should be tailored in a way that best suits the needs of the end users. Disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. When AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients, and to allow each to understand how decisions impacting patient care or access to care are made. While transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. In addition, AMA writes:
This bill appears aligned with AMA’s principles, specifically i) above.
- CONCERNS. Pending further analysis of the bill, the California Radiological Society (CRS) writes to express concern with the scope of the bill, and the impact it may have on these existing practices, including the necessity of such disclosures even if approved or signed off by a health care provider. CRS notes some of the complex ways AI may be utilized in many ways in the clinical setting, particularly radiology. According to CRS, AI can help triage imaging studies based on urgency. For instance, it can identify critical findings, such as a brain hemorrhage on a CT scan, and prioritize it for immediate review, thus potentially saving lives by reducing diagnostic turnaround times. CRS notes AI tools can also generate preliminary reports by identifying common findings in images. These reports are then often reviewed and finalized by a radiologist, significantly speeding up the reporting process. Finally, CRS indicates there may be clinical notes that might be generated partially by AI but still signed off by a human, or an email that may use AI to start a response but is then edited and completed by the physician.
- RELATED LEGISLATION.
- AB 3211 (Wicks) establishes the California Provenance, Authenticity, and Watermarking Standards. Among other requirements related to watermarking of digital content, AB 3211 requires a conversational AI system operating through an audio modality to verbally disclose that the conversational AI system receives synthetic content, as defined. It also requires a conversational AI system to obtain a user’s affirmative consent prior to beginning a conversation. AB 3211 is pending in Assembly Privacy and Consumer Protection Committee.
- AB 2905 (Low) and AB 2512 (Jim Patterson) are identical bills, and both include telephone calls made using an artificial voice under the Public Utilities Commission’s (PUC) current regulatory regime that applies to automatic dialing-announcing devices (“robocalls”). Both bills are pending in Assembly Communications and Conveyance Committee.
- AB 2811 (Lowenthal) requires an attorney to submit an affidavit with each document filed in a state or federal court that discloses the use of GenAI, as specified. AB 2811 is pending in the Assembly Judiciary Committee.
- SB 896 (Dodd) requires specified state agencies to take a number of actions related to analysis, risk assessment and use of GenAI. SB 896 includes requirements similar to those in this bill, i.e., it requires all state agency to disclose use of GenAI when interacting with the public and provide a person clear instructions how to directly communicate with a person from the state agency. SB 896 is pending in the Senate Governmental Organization Committee.
- PREVIOUS LEGISLATION. SB 313 (Dodd) contained similar requirements related to state agency interaction with the public as those in SB 896, as explained above. SB 313 was held on the suspense file of the Senate Appropriations Committee.
- POLICY COMMENTS.
- Alignment with Other Efforts. As discussed in the Background on recent state and federal activity, this bill aligns with existing frameworks promoting disclosure to the end user when AI is being used in potentially consequential ways. It does not appear misaligned with other state or federal efforts, though other bills under consideration by the Legislature address disclosure of the use of an AI system in other applications. Alignment
- Definitions. One of the significant challenges in any regulatory regime is clearly defining and specifying the entity or behavior that is being regulated. This bill is only one of a large number of bills proposing regulation of AI, and it only applies in the health care sphere. Definitions—for instance, of GenAI— will have to be reconciled across bills to have a coherent regulatory approach.
- Is A State-Level Regulation Appropriate For This Issue? Federal rules do not appear to address the particular situation raised in this bill, and the state is responsible for regulation of health facilities and professionals, making a state requirement appear reasonable. However, it is possible that federal rules or standards may supersede or conflict with these requirements at some future time.
- Is a Health Care-Specific Rule Appropriate For This Issue? As noted above, there are principles in AI best practices or regulation that can be applied more broadly. The application of AI in health care poses particular risks and no doubt there are some unique applications that can only be addressed by a health care-specific approach. However, disclosure to a consumer of the use or assistance of an AI system in interacting with any provider of a service may be something that can be effectively addressed through a broader lens of consumer protection.
- Where Do We Draw The Line Between GenAI-Assisted Content Generation And Automation Generally? Should GenAI uniquely trigger disclosure or is disclosure also appropriate for automated (non-GenAI) content? If a draft response is simply automated based on programmed rules and then reviewed by a clinician before being shared by a patient, should that also trigger a mandatory disclosure? As compared to a sophisticated but automated system that is not GenAI— is this particular application of GenAI different enough from automation, and/or does it pose a higher level of risk, that justifies the need for disclosure?
- How Is This Requirement Enforced? This bill does not create any enforcement regime or penalty for noncompliance, in fact, it specifies no liability or penalty for failure to comply with its requirements. The author explains the lack of “teeth” in enforcement is intended to send a market signal that the state expects private entities to take this approach and maintain a cautious balance between under-regulation and overregulation.
- Risk of Paralysis by Analysis. GenAI is a revolutionary technology with— it is safe to say— unimaginable potential for benefits and harms. It is also moving and evolving at light speed. The inherent complexity and difficulty of regulating the technology in a coherent and balanced way can be discouraging. However, choosing to do nothing while a powerful and volatile technology is loosed on a world that is still quite naïve to its incredible potential and multitudes of uses also has significant downside risk. In general, although this regulatory approach and other proposals raise a host of deep and novel questions, we cannot expect neat, satisfying answers as a condition of pursing regulation. We should expect our initial attempts at ensuring GenAI works well for humans will not be perfect, but that is a reason to collectively do our level best to work through the issues piece by piece, not to be paralyzed by waiting for a perfect, elegant solution that is unlikely to emerge.
- DOUBLE REFERRAL. This bill is double referred. Upon passage in this Committee, this bill will be re-referred to the Assembly Committee on Privacy and Consumer Protection.
- POLICY COMMENT. This bill includes a provision exempting providers from civil and criminal liability associated with the failure to provide the required notification. The provision is unnecessary as the bill creates no specific civil or criminal consequence for failure to comply with its provisions, it risks creating ambiguity about liability for actions outside of the failure to disclose, its nature as a blanket exemption is a bad precedent for nascent AI regulation and it risks creating a patchwork of liability provisions for different requirements that conflict with other broader, liability frameworks. California will be better-served with a coherent framework for liability associated with AI.
REGISTERED SUPPORT / OPPOSITION:
Support
None on file.
Opposition
None on file.
Analysis Prepared by: Lisa Murawski / HEALTH / (916) 319-2097
Discussed in Hearing
Assembly Floor
Senate Standing Committee on Appropriations
Senate Standing Committee on Health
Assembly Floor
Assembly Floor
Assembly Standing Committee on Health
Bill Author
Bill Co-Author(s):