SB 503: Health care services: artificial intelligence.
- Session Year: 2025-2026
- House: Senate
- Latest Version Date: 2026-09-30
Current Status:
Passed
(2026-09-30: Chaptered by Secretary of State. Chapter 857, Statutes of 2026.)
Introduced
In Committee
Awaiting First Chamber Vote
Passed First Chamber
In Committee
Awaiting Second Chamber Vote
Passed Second Chamber
Enacted
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require developers and deployers, as defined, of an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids decisionmaking related to diagnosis or treatment, known as a clinical decision support system, to make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk for biased impacts resulting from deployment of the system in health programs or activities. The bill would require developers to make a statement describing the intended uses and known or reasonably foreseeable risks associated with the use of the clinical decision support system and certain documentation available to deployers, as specified. The bill would require developers to make reasonable efforts to mitigate known or reasonably foreseeable risk for biased impacts resulting from use of the clinical decision support system in health programs or activities. The bill would require deployers to regularly monitor clinical decision support systems and take reasonable and proportionate steps to mitigate known or reasonably foreseeable risk of biased impacts. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require developers and deployers, as defined, of an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids decisionmaking related to diagnosis or treatment, known as a clinical decision support system, to make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk for biased impacts resulting from deployment of the system in health programs or activities. The bill would require developers to make a statement describing the intended uses and known or reasonably foreseeable risks associated with the use of the clinical decision support system and certain documentation available to deployers, as specified. The bill would require developers to make reasonable efforts to mitigate known or reasonably foreseeable risk for biased impacts resulting from use of the clinical decision support system in health programs or activities. The bill would require deployers to regularly monitor clinical decision support systems and take reasonable and proportionate steps to mitigate known or reasonably foreseeable risk of biased impacts. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require developers and deployers of artificial intelligence systems deployers, as defined, of an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids decisionmaking related to diagnosis or treatment, known as a clinical decision support system, to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk for biased impacts in the systems outputs resulting from use deployment of the system in health programs or activities. The bill would require developers to make a statement describing the intended uses and known or reasonably foreseeable risks associated with the use of the clinical decision support system and certain documentation available to deployers, as specified. The bill would require developers and deployers to make reasonable efforts to mitigate the known or reasonably foreseeable risk for biased impacts in the systems outputs resulting from use of the clinical decision support system in health programs or activities. The bill would require deployers to regularly monitor these artificial intelligence clinical decision support systems and take reasonable and proportionate steps to mitigate any bias that may occur. known or reasonably foreseeable risk of biased impacts. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer. The bill would specify that the department is not required to independently inspect, test, or evaluate the functionality of an artificial intelligence system. The bill would require, beginning January 1, 2027, developers to provide a report identifying compliance efforts with the above-described provisions to the department before making an artificial intelligence system commercially or publicly available to a deployer, as specified. The bill would require deployers, beginning January 1, 2027, to annually provide the department with a report identifying their efforts to comply with identification, mitigation, and monitoring requirements established pursuant to these provisions. The bill would require the department to make these reports available on its internet website.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require developers and deployers of artificial intelligence systems to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require developers and deployers to make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require deployers to regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer. The bill would require, beginning January 1, 2030, and at least annually thereafter, a developer to submit their artificial intelligence systems to an independent third-party auditor to assess whether the developer has complied with the above-described provisions. The bill would require developers to make a summary of the audit publicly available on its internet website. The bill would specify that the department is not required to independently inspect, test, or evaluate the functionality of an artificial intelligence system. The bill would require, beginning January 1, 2027, developers to provide a report identifying compliance efforts with the above-described provisions to the department before making an artificial intelligence system commercially or publicly available to a deployer, as specified. The bill would require deployers, beginning January 1, 2027, to annually provide the department with a report identifying their efforts to comply with identification, mitigation, and monitoring requirements established pursuant to these provisions. The bill would require the department to make these reports available on its internet website.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require developers and deployers of artificial intelligence systems and health facilities, clinics, physicians offices, or offices of a group practice to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require developers and deployers to make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require deployers to regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer. The bill would require, beginning January 1, 2030, and at least annually thereafter, a developer to submit their artificial intelligence systems to an independent third-party auditor to assess whether the developer has complied with the above-described provisions. The bill would require developers to make a summary of the audit publicly available on its internet website.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require developers of artificial intelligence systems, as defined, systems and health facilities, clinics, physicians offices, or offices of a group practice to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking and or health care resource allocation that are known or have a reasonably foreseeable risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require developers and deployers to make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require deployers to regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.This provider.
This bill would require developers of patient care decision support tools, artificial intelligence systems, as defined, and health facilities, clinics, physicians offices, or offices of a group practice to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic. artificial intelligence systems used to support clinical decisionmaking and health care resource allocation that are known or have a reasonably foreseeable risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require, for each patient care decision support tool, require developers and deployers to make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tools use for biased impacts in the systems outputs resulting from use of the system in health programs or activities. The bill would require developers ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least every 3 years. deployers to regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. Existing law establishes the Department of Health Care Access and Information to oversee and administer various health programs. Existing law establishes within the Government Operations Agency the Department of Technology, which is supervised by the Director of Technology. Existing law authorizes the director and the department to exercise various powers in creating and managing the information technology policy of the state.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require the Department of Health Care Access and Information and the Department of Technology to establish an advisory board related to the use of artificial intelligence in health care services. The bill would require the advisory board to perform specified duties, including, but not limited to, developing a standardized testing system with criteria for developers to test AI models or AI systems for biased impacts. The bill would require developers of AI models or AI systems, in conjunction with health facilities, clinics, physicians offices, or offices of a group practice, to test for biased impacts in the outputs produced by the specified AI model or AI system based on the health facilitys patient population, as specified. The bill would authorize developers to use the standardized testing system developed by the board to certify their AI models or AI systems.This bill would require developers of patient care decision support tools, as defined, and health facilities, clinics, physicians offices, or offices of a group practice to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic. The bill would require, for each patient care decision support tool, developers and deployers to make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tools use in health programs or activities. The bill would require developers ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least every 3 years. The bill would specify that a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
Existing law provides for the licensure and regulation of health facilities and clinics by the State Department of Public Health. Existing law establishes the Department of Health Care Access and Information to oversee and administer various health programs. Existing law establishes within the Government Operations Agency the Department of Technology, which is supervised by the Director of Technology. Existing law authorizes the director and the department to exercise various powers in creating and managing the information technology policy of the state.
Existing law requires a health facility, clinic, physicians office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information, as defined, to ensure that those communications include both (1) a disclaimer that indicates to the patient that a communication was generated by generative artificial intelligence, as specified, and (2) clear instructions describing how a patient may contact a human health care provider, employee, or other appropriate person. Existing law exempts from this requirement a communication read and reviewed by a human licensed or certified health care provider.
This bill would require the Department of Health Care Access and Information and the Department of Technology to establish an advisory board related to the use of artificial intelligence in health care services. The bill would require the advisory board to perform specified duties, including, but not limited to, developing a standardized testing system with criteria for developers to test AI models or AI systems for biased impacts. The bill would require developers of AI models or AI systems, in conjunction with health facilities, clinics, physicians offices, or offices of a group practice, to test for biased impacts in the outputs produced by the specified AI model or AI system based on the health facilitys patient population, as specified. The bill would authorize developers to use the standardized testing system developed by the board to certify their AI models or AI systems.
Existing law establishes the State Department of Public Health. Existing law, the Sherman Food, Drug, and Cosmetic Law, requires the department to regulate the manufacture, sale, labeling, and advertising activities related to food, drugs, medical instruments, and cosmetics in conformity with the federal Food, Drug, and Cosmetic Act.This bill would state the intent of the Legislature to enact legislation regarding the regulation of medical software.Existing law establishes the State Department of Public Health. Existing law, the Sherman Food, Drug, and Cosmetic Law, requires the department to regulate the manufacture, sale, labeling, and advertising activities related to food, drugs, medical instruments, and cosmetics in conformity with the federal Food, Drug, and Cosmetic Act.
This bill would state the intent of the Legislature to enact legislation regarding the regulation of medical software.
(a)(1)Developers and deployers shall make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk of biased impacts resulting from deployment of the system in health programs or activities.
(2)Developers shall make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of the clinical decision support systems identified pursuant to paragraph (1).
(3)Developers shall make at least the following available to deployers for clinical decision support systems identified pursuant to paragraph (1):
(A)A statement describing the intended uses and known or reasonably foreseeable risks associated with use of the clinical decision support system.
(B)Documentation disclosing all of the following:
(i)High-level summaries of the type or types of data used to train the clinical decision support system, including a description of its demographic representativeness when demographic data is available.
(ii)How the clinical decision support system was evaluated for performance, limitations, and mitigation of biased impacts.
(iii)High-level summaries of data governance measures used to evaluate training datasets and measures used to determine the suitability of data sources and possible biases.
(iv)Intended benefits and expected outputs of the clinical decision support system.
(v)Any known or reasonably foreseeable risk of biased impacts and efforts to mitigate those risks.
(vi)Recommendations regarding how the clinical decision support system should be used and monitored, and how risks should be mitigated.
(vii)Any other documentation that is reasonably necessary to help deployers understand the outputs and monitor the performance of the clinical decision support system for biased impacts.
(4)In complying with paragraph (3), developers may adhere to nationally recognized or widely adopted industry standards that are developed through multistakeholder consensus, relevant to bias testing and artificial intelligence systems in health care, or provide the results from regularly conducted algorithmic impact assessments using industry-accepted methodologies, including information relevant to the requirements specified in subparagraph (B) of paragraph (3), and appropriate for the clinical decision support systems intended use, to the extent that those standards or assessments exist.
(5)Documentation required under paragraph (3) shall be made available to deployers upon request or at the time of initial sale, whichever is earlier, and, as applicable, upon release of material updates to the clinical decision support system.
(6)Paragraph (3) does not prohibit developers from providing deployers with additional documentation upon request or to satisfy a contractual or reporting obligation.
(7)A deployer that also serves as a developer of a clinical decision support system described in paragraph (1) is not required to generate documentation required by subparagraph (B) of paragraph (3) if the system is developed and deployed for internal use and information consistent with the documentation and transparency requirements applicable to developers pursuant to subparagraph (B) of paragraph (3) is readily available to the deployer.
(8)Deployers shall regularly monitor the clinical decision support systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)For the purposes of this section, the following definitions shall apply:
(1)Artificial intelligence has the same meaning as in Section 11546.45.5 of the Government Code.
(2)Biased impact means an adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(3)Clinic has the same meaning as defined in Section 1200 or 1200.1 of the Health and Safety Code.
(4)Clinical decision support system means an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decisionmaking related to timing of care, diagnosis, or treatment. It does not include systems that provide appointment management, such as booking, canceling, and rescheduling appointments, appointment reminders, patient education and previsit materials and preparation, and payment processing, to the extent the independent performance of these activities by the system does not require a professional license.
(5)Deployer means a health facility, clinic, physicians office, or office of a group practice that uses a clinical decision support system.
(6)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a clinical decision support system for commercial or public use.
(7)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500).
(8)Health facility has the same meaning as Section 1250 of the Health and Safety Code.
(9)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204 of the Health and Safety Code.
(10)Physicians office means an office of a physician in solo practice.
(11)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
(a)(1)Developers and deployers shall make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk of biased impacts resulting from deployment of the system in health programs or activities.
(2)Developers shall make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of the clinical decision support systems identified pursuant to paragraph (1).
(3)Developers shall make at least the following available to deployers for clinical decision support systems identified pursuant to paragraph (1):
(A)A statement describing the intended uses and known or reasonably foreseeable risks associated with use of the clinical decision support system.
(B)Documentation disclosing all of the following:
(i)High-level summaries of the type or types of data used to train the clinical decision support system, including a description of its demographic representativeness when demographic data is available.
(ii)How the clinical decision support system was evaluated for performance, limitations, and mitigation of biased impacts.
(iii)High-level summaries of data governance measures used to evaluate training datasets and measures used to determine the suitability of data sources and possible biases.
(iv)Intended benefits and expected outputs of the clinical decision support system.
(v)Any known or reasonably foreseeable risk of biased impacts and efforts to mitigate those risks.
(vi)Recommendations regarding how the clinical decision support system should be used and monitored, and how risks should be mitigated.
(vii)Any other documentation that is reasonably necessary to help deployers understand the outputs and monitor the performance of the clinical decision support system for biased impacts.
(4)In complying with paragraph (3), developers may adhere to nationally recognized or widely adopted industry standards that are developed through multistakeholder consensus, relevant to bias testing and artificial intelligence systems in health care, or provide the results from regularly conducted algorithmic impact assessments using industry-accepted methodologies, including information relevant to the requirements specified in subparagraph (B) of paragraph (3), and appropriate for the clinical decision support systems intended use, to the extent that those standards or assessments exist.
(5)Documentation required under paragraph (3) shall be made available to deployers upon request or at the time of initial sale, whichever is earlier, and, as applicable, upon release of material updates to the clinical decision support system.
(6)Paragraph (3) does not prohibit developers from providing deployers with additional documentation upon request or to satisfy a contractual or reporting obligation.
(7)A deployer that also serves as a developer of a clinical decision support system described in paragraph (1) is not required to generate documentation required by subparagraph (B) of paragraph (3) if the system is developed and deployed for internal use and information consistent with the documentation and transparency requirements applicable to developers pursuant to subparagraph (B) of paragraph (3) is readily available to the deployer.
(8)Deployers shall regularly monitor the clinical decision support systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)For the purposes of this section, the following definitions shall apply:
(1)Artificial intelligence has the same meaning as in Section 11546.45.5 of the Government Code.
(2)Biased impact means an adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(3)Clinic has the same meaning as defined in Section 1200 or 1200.1 of the Health and Safety Code.
(4)Clinical decision support system means an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decisionmaking related to timing of care, diagnosis, or treatment. It does not include systems that provide appointment management, such as booking, canceling, and rescheduling appointments, appointment reminders, patient education and previsit materials and preparation, and payment processing, to the extent the independent performance of these activities by the system does not require a professional license.
(5)Deployer means a health facility, clinic, physicians office, or office of a group practice that uses a clinical decision support system.
(6)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a clinical decision support system for commercial or public use.
(7)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500).
(8)Health facility has the same meaning as Section 1250 of the Health and Safety Code.
(9)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204 of the Health and Safety Code.
(10)Physicians office means an office of a physician in solo practice.
(11)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
(a)(1)Developers and deployers shall make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk of biased impacts resulting from deployment of the system in health programs or activities.
(2)Developers shall make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of the clinical decision support systems identified pursuant to paragraph (1).
(3)Developers shall make at least the following available to deployers for clinical decision support systems identified pursuant to paragraph (1):
(A)A statement describing the intended uses and known or reasonably foreseeable risks associated with use of the clinical decision support system.
(B)Documentation disclosing all of the following:
(i)High-level summaries of the type or types of data used to train the clinical decision support system, including a description of its demographic representativeness when demographic data is available.
(ii)How the clinical decision support system was evaluated for performance, limitations, and mitigation of biased impacts.
(iii)High-level summaries of data governance measures used to evaluate training datasets and measures used to determine the suitability of data sources and possible biases.
(iv)Intended benefits and expected outputs of the clinical decision support system.
(v)Any known or reasonably foreseeable risk of biased impacts and efforts to mitigate those risks.
(vi)Recommendations regarding how the clinical decision support system should be used and monitored, and how risks should be mitigated.
(vii)Any other documentation that is reasonably necessary to help deployers understand the outputs and monitor the performance of the clinical decision support system for biased impacts.
(4)In complying with paragraph (3), developers may adhere to nationally recognized or widely adopted industry standards that are developed through multistakeholder consensus, relevant to bias testing and artificial intelligence systems in health care, or provide the results from regularly conducted algorithmic impact assessments using industry-accepted methodologies, including information relevant to the requirements specified in subparagraph (B) of paragraph (3), and appropriate for the clinical decision support systems intended use, to the extent that those standards or assessments exist.
(5)Documentation required under paragraph (3) shall be made available to deployers upon request or at the time of initial sale, whichever is earlier, and, as applicable, upon release of material updates to the clinical decision support system.
(6)Paragraph (3) does not prohibit developers from providing deployers with additional documentation upon request or to satisfy a contractual or reporting obligation.
(7)A deployer that also serves as a developer of a clinical decision support system described in paragraph (1) is not required to generate documentation required by subparagraph (B) of paragraph (3) if the system is developed and deployed for internal use and information consistent with the documentation and transparency requirements applicable to developers pursuant to subparagraph (B) of paragraph (3) is readily available to the deployer.
(8)Deployers shall regularly monitor the clinical decision support systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)For the purposes of this section, the following definitions shall apply:
(1)Artificial intelligence has the same meaning as in Section 11546.45.5 of the Government Code.
(2)Biased impact means an adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(3)Clinic has the same meaning as defined in Section 1200 or 1200.1 of the Health and Safety Code.
(4)Clinical decision support system means an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decisionmaking related to timing of care, diagnosis, or treatment. It does not include systems that provide appointment management, such as booking, canceling, and rescheduling appointments, appointment reminders, patient education and previsit materials and preparation, and payment processing, to the extent the independent performance of these activities by the system does not require a professional license.
(5)Deployer means a health facility, clinic, physicians office, or office of a group practice that uses a clinical decision support system.
(6)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a clinical decision support system for commercial or public use.
(7)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500).
(8)Health facility has the same meaning as Section 1250 of the Health and Safety Code.
(9)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204 of the Health and Safety Code.
(10)Physicians office means an office of a physician in solo practice.
(11)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
SECTION 1.Section 1339.76 is added to the Health and Safety Code, to read:1339.76.(a)(1)Developers and deployers shall have an ongoing duty to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the systems outputs resulting from use of the system in health programs or activities.(2)Developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the systems identified pursuant to paragraph (1) in health programs or activities.(3)Deployers shall regularly monitor these artificial intelligence systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate any bias that may occur.(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.(c)(1)The department shall not be required to independently inspect, test, or evaluate the functionality of an artificial intelligence system. Developers and deployers shall be responsible for ensuring that the artificial intelligence systems comply with the requirements set forth in subdivision (a). Developers and deployers shall maintain documentation of their respective compliance efforts.(2)Beginning January 1, 2027, a developer shall provide the department with a report identifying their efforts to comply with the requirements described in subdivision (a) for artificial intelligence systems, prior to the developer making a system commercially or publicly available to a deployer to be used to support clinical decisionmaking or health care resource allocation. A developer shall also provide the department an updated report identifying their efforts to comply with the requirements described in subdivision (a) for each substantial update of such a system prior to that systems initial deployment. Developer reporting obligations apply only prior to deployment and do not extend to postdeployment use.(3)Beginning January 1, 2027, and annually thereafter, deployers shall provide the department with a report identifying their efforts to comply with the requirements described in subdivision (a).(4)The department shall make the reports available on the departments internet website.(d)For purposes of this section, the following definitions apply:(1)Biased impact means an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.(3)Deployer means a health facility, clinic, physicians office, or office of a group practice that uses an artificial intelligence system to support clinical decisionmaking or health care resource allocation.(4)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an artificial intelligence system for commercial or public use to support clinical decisionmaking or health care resource allocation.(5)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.(6)Health facility has the same meaning as Section 1250.(7)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.(8)Artificial intelligence has the same meaning as in Section 11546.45.5 of the Government Code.(9)Physicians office means an office of a physician in solo practice.(10)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.(e)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.(a)(1)Developers and deployers shall have an ongoing duty to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the systems outputs resulting from use of the system in health programs or activities.
(2)Developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the systems identified pursuant to paragraph (1) in health programs or activities.
(3)Deployers shall regularly monitor these artificial intelligence systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate any bias that may occur.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)(1)Beginning January 1, 2030, and at least annually thereafter, a developer shall submit their artificial intelligence systems to an independent third-party auditor to assess whether the developer has complied with their duties pursuant to subdivision (a).(2)A developer subject to paragraph (1) shall make a high-level summary of the results of an audit required by that paragraph publicly available, at no cost, on the developers internet website.(c)(1)The department shall not be required to independently inspect, test, or evaluate the functionality of an artificial intelligence system. Developers and deployers shall be responsible for ensuring that the artificial intelligence systems comply with the requirements set forth in subdivision (a). Developers and deployers shall maintain documentation of their respective compliance efforts.
(2)Beginning January 1, 2027, a developer shall provide the department with a report identifying their efforts to comply with the requirements described in subdivision (a) for artificial intelligence systems, prior to the developer making a system commercially or publicly available to a deployer to be used to support clinical decisionmaking or health care resource allocation. A developer shall also provide the department an updated report identifying their efforts to comply with the requirements described in subdivision (a) for each substantial update of such a system prior to that systems initial deployment. Developer reporting obligations apply only prior to deployment and do not extend to postdeployment use.
(3)Beginning January 1, 2027, and annually thereafter, deployers shall provide the department with a report identifying their efforts to comply with the requirements described in subdivision (a).
(4)The department shall make the reports available on the departments internet website.
(d)For purposes of this section, the following definitions apply:
(1)Biased impact means an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.
(3)Deployer means a health facility, clinic, physicians office, or office of a group practice that uses an artificial intelligence system to support clinical decisionmaking or health care resource allocation.
(4)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an artificial intelligence system for commercial or public use to support clinical decisionmaking or health care resource allocation.
(5)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(6)Health facility has the same meaning as Section 1250.
(7)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.
(8)Artificial intelligence has the same meaning as in Section 11546.45.5 of the Government Code.
(9)Physicians office means an office of a physician in solo practice.
(10)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(e)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
(a)(1)Developers of artificial intelligence systems, and health facilities, clinics, physicians offices, or offices of a group practice, and deployers shall have an ongoing duty to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the systems outputs resulting from use of the system in health programs or activities.
(2)Developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the system systems identified pursuant to paragraph (1) in health programs or activities.
(3)Deployers shall regularly monitor these artificial intelligence systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate any bias that may occur.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)(1)Beginning January 1, 2030, and at least annually thereafter, a developer shall submit their artificial intelligence systems to an independent third-party auditor to assess whether the developer has complied with their duties pursuant to subdivision (a).
(2)A developer subject to paragraph (1) shall make a high-level summary of the results of an audit required by that paragraph publicly available, at no cost, on the developers internet website.
(c)(d)For purposes of this section, the following definitions apply:
(1)Biased impact means an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.
(3)Deployer means a person, partnership, state or local governmental agency, corporation, or developer health facility, clinic, physicians office, or office of a group practice that uses an artificial intelligence system to support clinical decisionmaking or health care resource allocation.
(4)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an artificial intelligence system for commercial or public use to support clinical decisionmaking or health care resource allocation.
(5)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(6)Health facility has the same meaning as Section 1250.
(7)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.
(8)Artificial intelligence has the same meaning as in Section 11546.45.5 of the Government Code.
(9)Physicians office means an office of a physician in solo practice.
(10)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d)(e)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
(a)(1)Developers of artificial intelligence systems, and health facilities, clinics, physicians offices, or offices of a group practice, shall have an ongoing duty to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking and or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the systems outputs resulting from use of the system in health programs or activities.
(2)Developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities.
(3)Deployers shall regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)For purposes of this section, the following definitions apply:
(1)Biased impact means an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.
(3)Deployer means a person, partnership, state or local governmental agency, corporation, or developer that uses an artificial intelligence system to support clinical decisionmaking and or health care resource allocation.
(4)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an artificial intelligence system for commercial or public use to support clinical decisionmaking and or health care resource allocation.
(5)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(6)Health facility has the same meaning as Section 1250.
(7)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.
(8)Artificial intelligence system means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments. intelligence has the same meaning as in Section 11546.45.5 of the Government Code.
(9)Physicians office means an office of a physician in solo practice.
(10)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
(a)(1)Developers of patient care decision support tools, artificial intelligence systems, and health facilities, clinics, physicians offices, or offices of a group practice, shall have an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic. artificial intelligence systems used to support clinical decisionmaking and health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the systems outputs resulting from use of the system in health programs or activities.
(2)For each patient care decision support tool, developers and deployers shall make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tools use in its health programs or activities.(3)Developers shall ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least once every three years.(2)Developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the systems outputs resulting from use of the system in health programs or activities.
(3)Deployers shall regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)For purposes of this section, the following definitions apply:
(1)Biased impact means an unintended impact adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.
(3)Deployer means a person, partnership, state or local governmental agency, corporation, or developer that uses a patient care decision support tool. an artificial intelligence system to support clinical decisionmaking and health care resource allocation.
(4)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a patient care decision support tool. an artificial intelligence system for commercial or public use to support clinical decisionmaking and health care resource allocation.
(5)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(6)Health facility has the same meaning as Section 1250.
(7)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.
(8)Patient care decision support tool means any automated or nonautomated tool, mechanism, method, technology, or combination thereof used by health facilities, clinics, physicians offices, or offices of a group practice to support clinical decisionmaking in its health programs or activities.(8)Artificial intelligence system means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
(9)Physicians office means an office of a physician in solo practice.
(10)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d)This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
(a)(1)Developers of patient care decision support tools, and health facilities, clinics, physicians offices, or offices of a group practice, shall have an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic.
(2)For each patient care decision support tool, developers and deployers shall make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tools use in its health programs or activities.
(3)Developers shall ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least once every three years.
(b)For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)For purposes of this section, the following definitions apply:
(1)Biased impact means an unintended impact on an individual based on their protected characteristics.
(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.
(3)Deployer means a person, partnership, state or local governmental agency, corporation, or developer that uses a patient care decision support tool.
(4)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a patient care decision support tool.
(5)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(6)Health facility has the same meaning as Section 1250.
(7)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.
(8)Patient care decision support tool means any automated or nonautomated tool, mechanism, method, technology, or combination thereof used by health facilities, clinics, physicians offices, or offices of a group practice to support clinical decisionmaking in its health programs or activities.
(9)Physicians office means an office of a physician in solo practice.
(10)Protected characteristic means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
SECTION 1.Section 1339.76 is added to the Health and Safety Code, to read:1339.76.(a)The Department of Health Care Access and Information and the Department of Technology shall establish an advisory board related to the use of artificial intelligence (AI) in health care services.(b)The advisory board shall do all of the following:(1)Develop best practices for the use of AI models or AI systems by a health facility, clinic, physicians office, or office of a group practice that uses AI in its provision of health care services.(2)Develop a standardized testing system with criteria for developers to test AI models or AI systems for biased impacts.(3)Establish a statewide certificate that can be used to confirm that a developers version or release of their AI model or AI system meets standards set by the advisory board pursuant to the testing for biased impacts described in paragraph (2).(c)(1)Developers of AI models or AI systems, in conjunction with health facilities, clinics, physicians offices, or offices of a group practice, shall test for biased impacts in the outputs produced by the specified AI model or AI system based on the health facilitys patient population.(2)Developers shall use an existing testing system designated by the advisory board until the advisory board has developed its standardized testing system described in paragraph (2) of subdivision (b). After the advisory board has developed its testing system, developers may alternatively use the boards testing system.(3)After the advisory board has created the certification described in paragraph (3) of subdivision (b), developers may use the advisory boards standardized testing system to certify their AI models or AI systems.(d)For purposes of this section, the following definitions apply:(1)Biased impact means an unintended impact on an individual based on their protected characteristics.(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.(3)(A)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an AI model or AI system that generates an output that can influence physical or virtual environments.(B)A deployer means a person, partnership, state or local governmental agency, corporation, or developer that uses a covered AI model or AI system that generates an output that can influence physical or virtual environments.(C)A person, partnership, state or local governmental agency, or corporation may be a developer and a deployer if the person or entity both designs, codes, substantially modifies, or otherwise produces an AI model or AI system that generates an output that can influence physical or virtual environments and also uses a covered AI model or AI system that generates an output that can influence physical or virtual environments.(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.(5)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.(6)Physicians office means an office of a physician in solo practice.(a)The Department of Health Care Access and Information and the Department of Technology shall establish an advisory board related to the use of artificial intelligence (AI) in health care services.
(b)The advisory board shall do all of the following:
(1)Develop best practices for the use of AI models or AI systems by a health facility, clinic, physicians office, or office of a group practice that uses AI in its provision of health care services.
(2)Develop a standardized testing system with criteria for developers to test AI models or AI systems for biased impacts.
(3)Establish a statewide certificate that can be used to confirm that a developers version or release of their AI model or AI system meets standards set by the advisory board pursuant to the testing for biased impacts described in paragraph (2).
(c)(1)Developers of AI models or AI systems, in conjunction with health facilities, clinics, physicians offices, or offices of a group practice, shall test for biased impacts in the outputs produced by the specified AI model or AI system based on the health facilitys patient population.
(2)Developers shall use an existing testing system designated by the advisory board until the advisory board has developed its standardized testing system described in paragraph (2) of subdivision (b). After the advisory board has developed its testing system, developers may alternatively use the boards testing system.
(3)After the advisory board has created the certification described in paragraph (3) of subdivision (b), developers may use the advisory boards standardized testing system to certify their AI models or AI systems.
(d)For purposes of this section, the following definitions apply:
(1)Biased impact means an unintended impact on an individual based on their protected characteristics.
(2)Clinic has the same meaning as defined in Section 1200 or 1200.1.
(3)(A)Developer means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an AI model or AI system that generates an output that can influence physical or virtual environments.
(B)A deployer means a person, partnership, state or local governmental agency, corporation, or developer that uses a covered AI model or AI system that generates an output that can influence physical or virtual environments.
(C)A person, partnership, state or local governmental agency, or corporation may be a developer and a deployer if the person or entity both designs, codes, substantially modifies, or otherwise produces an AI model or AI system that generates an output that can influence physical or virtual environments and also uses a covered AI model or AI system that generates an output that can influence physical or virtual environments.
(4)Health care provider means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(5)Office of a group practice means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or not-for-profit corporation licensed according to subdivision (a) of Section 1204.
(6)Physicians office means an office of a physician in solo practice.
SECTION 1.It is the intent of the Legislature to enact legislation regarding the regulation of medical software.It is the intent of the Legislature to enact legislation regarding the regulation of medical software.
|
Date
|
Status
|
|---|---|
| Sep 30, 2026 | Chaptered by Secretary of State. Chapter 857, Statutes of 2026. |
| Sep 30, 2026 | Approved by the Governor. |
| Aug 30, 2026 | Enrolled and presented to the Governor at 6 p.m. |
| Aug 25, 2026 | Assembly amendments concurred in. (Ayes 39. Noes 0. Page 5739.) Ordered to engrossing and enrolling. |
| Aug 25, 2026 | Assembly amendments concurred in. (Ayes 39. Noes 0.) Ordered to engrossing and enrolling. |
| Aug 24, 2026 | In Senate. Concurrence in Assembly amendments pending. |
| Aug 24, 2026 | Read third time. Passed. (Ayes 70. Noes 1. Page 6384.) Ordered to the Senate. |
| Aug 24, 2026 | Read third time. Passed. Ordered to the Senate. |
| Aug 19, 2026 | Read third time and amended. |
| Aug 19, 2026 | Ordered to third reading. |
| Aug 12, 2026 | Ordered to third reading. |
| Aug 12, 2026 | From inactive file. |
| Aug 11, 2026 | Notice of intention to remove from inactive file given by Assembly Member Wilson. |
| Sep 10, 2025 | Ordered to inactive file on request of Assembly Member Aguiar-Curry. |
| Sep 4, 2025 | Ordered to third reading. |
| Sep 4, 2025 | Read third time and amended. |
| Sep 2, 2025 | Read second time. Ordered to third reading. |
| Aug 29, 2025 | From committee: Do pass. (Ayes 15. Noes 0.) (August 29). |
| Aug 20, 2025 | August 20 set for first hearing. Placed on APPR. suspense file. |
| Jul 17, 2025 | Read second time and amended. Re-referred to Com. on APPR. |
| Jul 17, 2025 | From committee: Do pass as amended and re-refer to Com. on APPR. (Ayes 13. Noes 0.) (July 16). |
| Jul 17, 2025 | Assembly Rule 63 suspended. (Ayes 49. Noes 15. Page 2578.) |
| Jul 10, 2025 | Read second time and amended. Re-referred to Com. on P. & C.P. |
| Jul 9, 2025 | From committee: Do pass as amended and re-refer to Com. on P. & C.P. (Ayes 16. Noes 0.) (July 8). |
| Jun 30, 2025 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on HEALTH. |
| Jun 24, 2025 | July 1 hearing postponed by committee. |
| Jun 5, 2025 | Referred to Coms. on HEALTH and P. & C.P. |
| May 29, 2025 | In Assembly. Read first time. Held at Desk. |
| May 29, 2025 | Read third time. Passed. (Ayes 38. Noes 0. Page 1327.) Ordered to the Assembly. |
| May 27, 2025 | Ordered to special consent calendar. |
| May 23, 2025 | Read second time. Ordered to third reading. |
| May 23, 2025 | From committee: Do pass. (Ayes 6. Noes 0. Page 1202.) (May 23). |
| May 16, 2025 | Set for hearing May 23. |
| May 12, 2025 | May 12 hearing: Placed on APPR. suspense file. |
| May 2, 2025 | Set for hearing May 12. |
| Apr 30, 2025 | From committee: Do pass and re-refer to Com. on APPR. (Ayes 13. Noes 0. Page 941.) (April 29). Re-referred to Com. on APPR. |
| Apr 23, 2025 | Re-referred to Com. on JUD. |
| Apr 22, 2025 | Set for hearing April 29 in JUD. pending receipt. |
| Apr 10, 2025 | Re-referred to Com. on RLS. |
| Apr 10, 2025 | Withdrawn from committee. |
| Apr 10, 2025 | Read second time and amended. Re-referred to Com. on G.O. |
| Apr 10, 2025 | From committee: Do pass as amended and re-refer to Com. on G.O. (Ayes 10. Noes 0. Page 737.) (April 9). |
| Apr 2, 2025 | Set for hearing April 9. |
| Apr 2, 2025 | Re-referred to Coms. on HEALTH and G.O. |
| Mar 25, 2025 | From committee with author's amendments. Read second time and amended. Re-referred to Com. on RLS. |
| Feb 26, 2025 | Referred to Com. on RLS. |
| Feb 20, 2025 | From printer. May be acted upon on or after March 22. |
| Feb 19, 2025 | Introduced. Read first time. To Com. on RLS. for assignment. To print. |
|
Date
|
Description
|
Location
|
Votes
|
|
|---|---|---|---|---|
| Aug 25, 2026 | Unfinished Business SB503 Weber Pierson et al. Concurrence | Senate Floor |
Passed |
|
| Aug 24, 2026 | SB 503 Weber Pierson Senate Third Reading By Wilson | Assembly Floor |
Passed |
|
| Aug 29, 2025 | Do pass. | Assembly Standing Committee on Appropriations |
Passed |
|
| Jul 16, 2025 | Do pass as amended and be re-referred to the Committee on [Appropriations] | Assembly Standing Committee on Privacy and Consumer Protection |
Passed |
|
| Jul 8, 2025 | Do pass as amended and be re-referred to the Committee on [Privacy and Consumer Protection] | Assembly Standing Committee on Health |
Passed |
|
| May 29, 2025 | Special Consent SB503 Weber Pierson et al. | Senate Floor |
Passed |
|
| May 23, 2025 | Do pass | Senate Standing Committee on Appropriations |
Passed |
|
| May 12, 2025 | Placed on suspense file | Senate Standing Committee on Appropriations |
Passed |
|
| Apr 29, 2025 | Do pass, but first be re-referred to the Committee on [Appropriations] | Senate Standing Committee on Judiciary |
Passed |
|
| Apr 9, 2025 | Do pass as amended, but first amend, and re-refer to the Committee on [Governmental Organization] | Senate Standing Committee on Health |
Passed |
This list shows positions taken on this bill by organizations on the date displayed, which is linked to the original source.
Positions by individuals are not included.
|
Organization
|
Position
|
Date
|
|---|---|---|
| AdvaMed | ** Against | |
| Advanced Medical Technology Association | ** Against | |
| America’s Physician Groups | ** Support | |
| ATA Action | ** Against | |
| Biocom | ** Against | |
| California Association Of Health Plans | ** Support | |
| California Hospital Association (CHA) | ** Support | |
| California Life Sciences | ** Against | |
| California Medical Association | ** Support | |
| California Radiological Society | ** Support | |
| Kaiser Permanente | ** Support | |
| Oakland Privacy | ** Support | |
| OCHIN INC. | ** Support | |
| Technet | ** Against | |
| University of California | ** Support |
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 503 |
UNFINISHED BUSINESS
Bill No: SB 503
Author: Weber Pierson (D), et al.
Amended: 8/19/26
Vote: 21
SENATE HEALTH COMMITTEE: 10-0, 4/9/25
AYES: Menjivar, Durazo, Gonzalez, Grove, Limón, Padilla, Richardson, Rubio, Weber Pierson, Wiener
NO VOTE RECORDED: Valladares
SENATE JUDICIARY COMMITTEE: 13-0, 4/29/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SENATE FLOOR: 38-0, 5/29/25
AYES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Cervantes, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
NO VOTE RECORDED: Limón, Reyes
ASSEMBLY FLOOR: 70-1, 8/24/26 - See last page for vote
SUBJECT: Health care services: artificial intelligence
SOURCE: Author
DIGEST:
This bill (1) requires developers of clinical decision support systems, as defined, to make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of these systems; and (2) requires developers to provide deployers of systems identified as presenting a risk of biased impacts with specified information, including documentation of how the system was evaluated for performance, limitations, and mitigation of biased impacts.
Assembly Amendments
revised and recast the provisions of this bill, including by changing terminology from patient care decision support tools to clinical decision support systems to conform with other health care AI bills, deleted a provision requiring the tools to be tested for bias every three years, added a requirement that developers make certain information available to users of clinical decision support systems, and made other technical, clarifying, and conforming changes.
ANALYSIS:
Existing law:
- Establishes the Department of Health Care Access and Information (HCAI), and designates HCAI as the state agency designated to collect health facility data for use by all state agencies, including various financial data reports. [Health and Safety Code (HSC) §128730, et seq.]
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative AI to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative AI, and clear instructions on how to contact a human person. Defines “artificial intelligence,” for these purposes, as an engineered or machine-based system that varies in in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments. [HSC §1339.75]
- Requires the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, and state agency. Defines “high-risk automated decision system” as an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. [Government Code (GOV) §11546.45.5]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [Civil Code (CIV) §1798.100, et seq.]
- Establishes the Unruh Civil Rights Act that entitles all persons within the jurisdiction of this state to full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status. [CIV §51(b)]
This bill:
- Requires developers and deployers, as each of these are defined, to make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk of biased impacts resulting from the deployment of the system in health programs or activities.
- Requires developers to make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of the clinical decision support systems identified pursuant to 1) above.
- Requires developers to provide deployers of clinical decision support systems identified pursuant to 1) above a statement describing the intended uses and known or reasonably foreseeable risks associated with use of the clinical decision support system, and documentation disclosing all of the following:
- High-level summaries of the type or types of data used to train the clinical decision support system, including a description of its demographic representativeness when demographic data is available;
- How the clinical decision support system was evaluated for performance, limitations, and mitigation of biased impacts;
- High-level summaries of data governance measures used to evaluate training datasets and measures used to determine the suitability of data sources and possible biases;
- Intended benefits and expected outputs of the clinical decision support system;
- Any known or reasonably foreseeable risk of biased impacts and efforts to mitigate those risks;
- Recommendations regarding how the clinical decision support system should be used and monitored, and how risks should be mitigated; and,
- Any other documentation that is reasonably necessary to help deployers understand the outputs and monitor the performance of the clinical decision support system for biased impacts.
- Permits developers, in complying with the disclosures in 3) above, to adhere to nationally recognized or widely adopted industry standards that are developed through multistakeholder consensus, relevant to bias testing and artificial intelligence systems in health care, or provide the results from regularly conducted algorithmic impact assessments using industry-accepted methodologies, including information that is relevant and appropriate for the clinical decision support system’s intended use, to the extent that those standards or assessments exist.
- Requires the documentation required under 3) above to be made available to deployers upon request or at the time of initial sale, whichever is earlier, and, as applicable, upon release of material updates to the clinical decision support system.
- Specifies that a deployer that also serves as a developer of a clinical decision support system is not required to generate documentation for 3) above if the system is developed and deployed for internal use and information consistent with 3) above is readily available to the deployer.
- Requires deployers to regularly monitor the clinical decision support systems identified pursuant to 1) above and to take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts.
- Defines various terms for purposes of this bill, including the following:
- “Biased impact,” means an adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics;
- “Clinical decision support system” means an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decisionmaking related to timing of care, diagnosis, or treatment. Excludes systems that provide appointment management, such as booking, canceling, and rescheduling appointments, appointment reminders, patient education and previsit materials and preparation, and payment processing, to the extent that the independent performance of these activities by the system does not require a professional license.
- “Developer” means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a clinical decision support system for commercial or public use.
- “Deployer” means a health facility, clinic, physician’s office, or office of a group practice that uses a clinical decision support system.
- “Protected characteristic” means any characteristic listed in the Unruh Civil Rights Act, as specified.
- Specifies that a person, partnership, state or local governmental agency, or corporation can be both a developer and a deployer.
Background
According to the author of this bill:
This bill is a crucial step toward ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. These technologies are becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill establishes a framework to identify and address biased outputs, ensuring that AI tools in healthcare are fair, accurate, and equitable for all patients. By taking a proactive approach, this bill helps protect patients from the effects of biased algorithms and ensures that technological advancements serve all communities equitably.
What is AI? In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
According to an article in MIT News, from the Massachusetts Institute of Technology, before the generative AI boom of the past few years, when people talked about AI, typically they were talking about machine-learning models that can learn to make a prediction based on data. For instance, such models are trained, using millions of examples, to predict whether a certain X-ray shows signs of a tumor or if a particular borrower is likely to default on a loan. Generative AI can be thought of as a machine-learning model that is trained to create new data, rather than making a prediction about a specific dataset. A generative AI system is one that learns to generate more objects that look like the data it was trained on. Generative AI systems such as ChatGPT are powerful because they are trained on extremely large datasets, which could potentially take advantage of nearly all the information on the internet.
According to an article published by CalTech Science Exchange, today’s generative AI models produce content that often is indistinguishable from that created by humans. Generative AI has successfully written news articles, created realistic artwork, and composed aesthetically pleasing music. While these applications sometimes make glaring mistakes (sometimes referred to as hallucinations), they are being used for many purposes, including in health care.
Generative AI in health care. According to a July 25, 2023 viewpoint article in the Journal of the American Medical Association, “Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients,” generative AI is being heralded in the medical field for its potential to ease the long-lamented burden of medical documentation by generating visit notes, treatment codes, and medical summaries. This article stated that to date, no court in the U.S. has considered the question of liability for medical injuries caused by relying on AI-generated information. The article warned that to minimize risks, even medical professionals must exercise caution when relying on AI-generated information, due to the “black-box” nature of how these systems generate output, because it may be impossible for the physician to independently evaluate the accuracy of the AI’s output. A March 2024 article in Implement Science stated that the utility and impact of generative AI in health care remain poorly understood, with concerns about ethical and legal implications, integration into healthcare service delivery, and workforce utilization. The American Medical Association published “Principles for Augmented Intelligence Development, Deployment, and Use,” which was approved by the American Medical Association (AMA) Board of Trustees on November 14, 2023. According to this AMA document, as the number of AI-enabled health care tools and systems continue to grow, these technologies must be designed, developed, and deployed in a manner that is ethical, equitable, responsible, and transparent. The AMA notes that while the FDA regulates AI-enabled medical devices, many types of AI-enabled technologies fall outside the scope of FDA oversight, including AI that may have clinical applications, such as some clinical decision support functions. The AMA report specifically looked at transparency in use of AI-enabled systems, and stated that it is essential that use of AI in health care be transparent to both physicians and patients, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. The AMA report states that when AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients. The AMA noted that while transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. According to the AMA, when AI is used in a manner, which directly impacts patient care, access to care, or medical decision-making, that use of AI should be disclosed and documented to both physicians and patients. The opportunity for a patient to request additional review from a licensed clinician should be made available upon request. The AMA policy also states that AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
Racial bias in algorithms and AI. Several studies have identified risks of racial bias in algorithms and AI. A October 2019 article in the journal Science, “Dissecting racial bias in an algorithm used to manage the health of populations,” found that Black patients assigned the same level of risk by the algorithm are sicker than White patients. The study estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. According to the study, bias occurs because the algorithm uses health costs as a proxy for health needs, and because less money is spent on Black patients who have the same level of need, the algorithm falsely concludes that Black patients are healthier than equally sick White patients. The journal NPJ Digital Medicine published a study in 2023, “Large language models propagate race-based medicine,” that assessed four large language models used in healthcare systems. The study found that all models had examples of perpetuating race-based medicine in their response to questions. Models were not always consistent in their response when asked the same question repeatedly, and that based on their findings, these large language models could potentially cause harm by perpetuating debunked, racist ideas.
Trustworthy AI. The federal Health and Human Services Agency (HHS) has developed a “playbook” for AI development with the goal of maintaining public trust by using AI solutions that are “ethical, effective, and secure,” according to the HHS Chief AI Officer. With regard to HHS agencies, principles have been established for trustworthy AI that “fosters public trust and confidence while protecting privacy, civil rights, civil liberties, and American values, consistent with applicable laws.” The playbook describes AI as whether the solution or system:
- Performs tasks under varying and unpredictable circumstances without significant human oversight, or can learn from experience and improvement performance when exposed to data sets;
- Uses computer software, physical hardware, or other technology to solve tasks that require human-like perception, thinking, planning, learning, communication, or physical action;
- Thinks or acts like a human, including the use of cognitive architecture or neural networks (e.g., developed to mimic the underlying mechanisms of the human mind);
- Relies on a set of techniques, including machine learning, to approximate a cognitive task; and,
- Is designed to act rationally by utilizing intelligent software or an embodied robot to achieve goals using perception, planning, reasoning, learning, communicating, decision-making, and action.
The HHS principles for use of trustworthy AI in government cover six areas: fair and impartial application; transparent and explainable data use; responsible and accountable governance and policies; robust and reliable outputs; safe and secure from risks; and, that privacy should be respected, data not used beyond its intended and stated use, and that it’s use is approved by the data owner or steward.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Assembly Appropriations Committee analysis, costs to the University of California (UC) of an unknown but potentially significant amount. According to UC Office of the President (UCOP), costs for vendor auditing will likely be passed on to UC health systems. UCOP also states the third-party audit requirement will disincentivize AI development within academic medical centers by adding substantial new cost for implementing internally-developed AI solutions. UCOP indicates these costs would reduce hospital revenue and create cost pressures for UC and the General Fund.
SUPPORT: (Verified 8/24/26)
America’s Physician Groups
California Association of Health Plans
California Medical Association
California Radiological Society
Kaiser Permanente
OCHIN Inc.
University of California
OPPOSITION: (Verified 8/24/26)
Advanced Medical Technology Association
ARGUMENTS IN SUPPORT: A coalition representing physicians, hospitals, physician groups, health plans, and other stakeholders, including the California Medical Association, Kaiser Permanente, and the California Association of Health Plans, submitted a letter of support to the most recent version of this bill. These supporters state that this bill would provide valuable information to health care deployers such as clinics, rural hospitals, independent physician practices, and others, before making significant technology/AI investments. Under this bill, developers would be required to provide information that will significantly help small health care providers to make informed decisions and better understand the strengths and limitations of tools they are considering using. This bill would equip deployers with necessary information to mitigate bias and risks of such tools and systems. These supporters state that AI holds extraordinary potential to transform and improve every aspect of health care, and this bill helps to level the playing field by ensuring all purchasers, regardless of size, have access to basic information necessary to determine whether a tool is appropriate for their patients and clinical workflows.
ARGUMENTS IN OPPOSITION: The Advanced Medical Technology Association (AdvaMed) submitted a letter of opposition to the most recent version of this bill. According to AdvaMed, unlike many other industries, the use of AI in medical technology is already subject to robust oversight and regulation by the FDA throughout the device’s lifecycle. FDA’s premarket review of AI-enabled devices includes an assessment of clinical relevant information, potential risk, the mitigation of unwanted bias, and performance validation on the device’s intended use and patient populations. Additionally, medical device manufacturers must follow strict federal requirements and regulations once devices are on the market, including monitoring and reporting of device malfunctions and serious adverse events. AdvaMed states that while they value the goal of this bill, they are concerned with the potential unintended consequences. AdvaMed’s concerns include the lack of clear definitions; that it overlooks the existing regulatory framework; and that it would establish duplicative California-specific requirements, and risks to confidentiality and intellectual property.
Assembly Floor: 70 1 8 Ayes: Addis, Aguiar-Curry, Ahrens, Alanis, Alvarez, Arambula, Ávila Farías, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Calderon, Caloza, Carrillo, Chen, Connolly, Davies, Dixon, Elhawary, Ellis, Flora, Fong, Gabriel, Garcia, Gipson, Jeff Gonzalez, Mark González, Haney, Harabedian, Hart, Irwin, Jackson, Kalra, Krell, Lee, Lowenthal, Muratsuchi, Nguyen, Ortega, Pacheco, Papan, Patel, Patterson, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Ransom, Celeste Rodriguez, Michelle Rodriguez, Rogers, Blanca Rubio, Sanchez, Schiavo, Schultz, Sharp-Collins, Solache, Soria, Stefani, Ta, Valencia, Wallis, Ward, Wicks, Wilson, Zbur, Rivas Noes: DeMaio No Vote Recorded: Castillo, Hadwick, Hoover, Johnson, Lackey, Macedo, McKinnor, Tangipa
Prepared by: Vincent Marchand / HEALTH / (916) 651-4111
11/5/26 15:28:10
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 503 |
UNFINISHED BUSINESS
Bill No: SB 503
Author: Weber Pierson (D), et al.
Amended: 8/19/26
Vote: 21
SENATE HEALTH COMMITTEE: 10-0, 4/9/25
AYES: Menjivar, Durazo, Gonzalez, Grove, Limón, Padilla, Richardson, Rubio, Weber Pierson, Wiener
NO VOTE RECORDED: Valladares
SENATE JUDICIARY COMMITTEE: 13-0, 4/29/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SENATE FLOOR: 38-0, 5/29/25
AYES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Cervantes, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
NO VOTE RECORDED: Limón, Reyes
ASSEMBLY FLOOR: Passed, 8/24/26
SUBJECT: Health care services: artificial intelligence
SOURCE: Author
DIGEST:
This bill (1) requires developers of clinical decision support systems, as defined, to make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of these systems; and, (2) requires developers to provide deployers of systems identified as presenting a risk of biased impacts with specified information, including documentation of how the system was evaluated for performance, limitations, and mitigation of biased impacts.
Assembly Amendments
revised and recast the provisions of this bill, including by changing terminology from patient care decision support tools to clinical decision support systems to conform with other health care AI bills, deleted a provision requiring the tools to be tested for bias every three years, added a requirement that developers make certain information available to users of clinical decision support systems, and made other technical, clarifying, and conforming changes.
ANALYSIS:
Existing law:
- Establishes the Department of Health Care Access and Information (HCAI), and designates HCAI as the state agency designated to collect health facility data for use by all state agencies, including various financial data reports. [Health and Safety Code (HSC) §128730, et seq.]
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative AI to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative AI, and clear instructions on how to contact a human person. Defines “artificial intelligence,” for these purposes, as an engineered or machine-based system that varies in in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments. [HSC §1339.75]
- Requires the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, and state agency. Defines “high-risk automated decision system” as an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. [Government Code (GOV) §11546.45.5]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [Civil Code (CIV) §1798.100, et seq.]
- Establishes the Unruh Civil Rights Act that entitles all persons within the jurisdiction of this state to full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status. [CIV §51(b)]
This bill:
- Requires developers and deployers, as each of these are defined, to make reasonable efforts to identify clinical decision support systems developed for use by deployers that are known or have a reasonably foreseeable risk of biased impacts resulting from the deployment of the system in health programs or activities.
- Requires developers to make reasonable efforts to mitigate the known or reasonably foreseeable risk of biased impacts resulting from the use of the clinical decision support systems identified pursuant to 1) above.
- Requires developers to provide deployers of clinical decision support systems identified pursuant to 1) above a statement describing the intended uses and known or reasonably foreseeable risks associated with use of the clinical decision support system, and documentation disclosing all of the following:
- High-level summaries of the type or types of data used to train the clinical decision support system, including a description of its demographic representativeness when demographic data is available;
- How the clinical decision support system was evaluated for performance, limitations, and mitigation of biased impacts;
- High-level summaries of data governance measures used to evaluate training datasets and measures used to determine the suitability of data sources and possible biases;
- Intended benefits and expected outputs of the clinical decision support system;
- Any known or reasonably foreseeable risk of biased impacts and efforts to mitigate those risks;
- Recommendations regarding how the clinical decision support system should be used and monitored, and how risks should be mitigated; and,
- Any other documentation that is reasonably necessary to help deployers understand the outputs and monitor the performance of the clinical decision support system for biased impacts.
- Permits developers, in complying with the disclosures in 3) above, to adhere to nationally recognized or widely adopted industry standards that are developed through multistakeholder consensus, relevant to bias testing and artificial intelligence systems in health care, or provide the results from regularly conducted algorithmic impact assessments using industry-accepted methodologies, including information that is relevant and appropriate for the clinical decision support system’s intended use, to the extent that those standards or assessments exist.
- Requires the documentation required under 3) above to be made available to deployers upon request or at the time of initial sale, whichever is earlier, and, as applicable, upon release of material updates to the clinical decision support system.
- Specifies that a deployer that also serves as a developer of a clinical decision support system is not required to generate documentation for 3) above if the system is developed and deployed for internal use and information consistent with 3) above is readily available to the deployer.
- Requires deployers to regularly monitor the clinical decision support systems identified pursuant to 1) above and to take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts.
- Defines various terms for purposes of this bill, including the following:
- “Biased impact,” means an adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics;
- “Clinical decision support system” means an artificial intelligence system that produces a prediction, classification, recommendation, evaluation, or analysis that aids clinical decisionmaking related to timing of care, diagnosis, or treatment. Excludes systems that provide appointment management, such as booking, canceling, and rescheduling appointments, appointment reminders, patient education and previsit materials and preparation, and payment processing, to the extent that the independent performance of these activities by the system does not require a professional license.
- “Developer” means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a clinical decision support system for commercial or public use.
- “Deployer” means a health facility, clinic, physician’s office, or office of a group practice that uses a clinical decision support system.
- “Protected characteristic” means any characteristic listed in the Unruh Civil Rights Act, as specified.
- Specifies that a person, partnership, state or local governmental agency, or corporation can be both a developer and a deployer.
Background
According to the author of this bill:
This bill is a crucial step toward ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. These technologies are becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill establishes a framework to identify and address biased outputs, ensuring that AI tools in healthcare are fair, accurate, and equitable for all patients. By taking a proactive approach, this bill helps protect patients from the effects of biased algorithms and ensures that technological advancements serve all communities equitably.
What is AI? In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
According to an article in MIT News, from the Massachusetts Institute of Technology, before the generative AI boom of the past few years, when people talked about AI, typically they were talking about machine-learning models that can learn to make a prediction based on data. For instance, such models are trained, using millions of examples, to predict whether a certain X-ray shows signs of a tumor or if a particular borrower is likely to default on a loan. Generative AI can be thought of as a machine-learning model that is trained to create new data, rather than making a prediction about a specific dataset. A generative AI system is one that learns to generate more objects that look like the data it was trained on. Generative AI systems such as ChatGPT are powerful because they are trained on extremely large datasets, which could potentially take advantage of nearly all the information on the internet.
According to an article published by CalTech Science Exchange, today’s generative AI models produce content that often is indistinguishable from that created by humans. Generative AI has successfully written news articles, created realistic artwork, and composed aesthetically pleasing music. While these applications sometimes make glaring mistakes (sometimes referred to as hallucinations), they are being used for many purposes, including in health care.
Generative AI in health care. According to a July 25, 2023 viewpoint article in the Journal of the American Medical Association, “Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients,” generative AI is being heralded in the medical field for its potential to ease the long-lamented burden of medical documentation by generating visit notes, treatment codes, and medical summaries. This article stated that to date, no court in the U.S. has considered the question of liability for medical injuries caused by relying on AI-generated information. The article warned that to minimize risks, even medical professionals must exercise caution when relying on AI-generated information, due to the “black-box” nature of how these systems generate output, because it may be impossible for the physician to independently evaluate the accuracy of the AI’s output. A March 2024 article in Implement Science stated that the utility and impact of generative AI in health care remain poorly understood, with concerns about ethical and legal implications, integration into healthcare service delivery, and workforce utilization. The American Medical Association published “Principles for Augmented Intelligence Development, Deployment, and Use,” which was approved by the American Medical Association (AMA) Board of Trustees on November 14, 2023. According to this AMA document, as the number of AI-enabled health care tools and systems continue to grow, these technologies must be designed, developed, and deployed in a manner that is ethical, equitable, responsible, and transparent. The AMA notes that while the FDA regulates AI-enabled medical devices, many types of AI-enabled technologies fall outside the scope of FDA oversight, including AI that may have clinical applications, such as some clinical decision support functions. The AMA report specifically looked at transparency in use of AI-enabled systems, and stated that it is essential that use of AI in health care be transparent to both physicians and patients, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. The AMA report states that when AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients. The AMA noted that while transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. According to the AMA, when AI is used in a manner, which directly impacts patient care, access to care, or medical decision-making, that use of AI should be disclosed and documented to both physicians and patients. The opportunity for a patient to request additional review from a licensed clinician should be made available upon request. The AMA policy also states that AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
Racial bias in algorithms and AI. Several studies have identified risks of racial bias in algorithms and AI. A October 2019 article in the journal Science, “Dissecting racial bias in an algorithm used to manage the health of populations,” found that Black patients assigned the same level of risk by the algorithm are sicker than White patients. The study estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. According to the study, bias occurs because the algorithm uses health costs as a proxy for health needs, and because less money is spent on Black patients who have the same level of need, the algorithm falsely concludes that Black patients are healthier than equally sick White patients. The journal NPJ Digital Medicine published a study in 2023, “Large language models propagate race-based medicine,” that assessed four large language models used in healthcare systems. The study found that all models had examples of perpetuating race-based medicine in their response to questions. Models were not always consistent in their response when asked the same question repeatedly, and that based on their findings, these large language models could potentially cause harm by perpetuating debunked, racist ideas.
Trustworthy AI. The federal Health and Human Services Agency (HHS) has developed a “playbook” for AI development with the goal of maintaining public trust by using AI solutions that are “ethical, effective, and secure,” according to the HHS Chief AI Officer. With regard to HHS agencies, principles have been established for trustworthy AI that “fosters public trust and confidence while protecting privacy, civil rights, civil liberties, and American values, consistent with applicable laws.” The playbook describes AI as whether the solution or system:
- Performs tasks under varying and unpredictable circumstances without significant human oversight, or can learn from experience and improvement performance when exposed to data sets;
- Uses computer software, physical hardware, or other technology to solve tasks that require human-like perception, thinking, planning, learning, communication, or physical action;
- Thinks or acts like a human, including the use of cognitive architecture or neural networks (e.g., developed to mimic the underlying mechanisms of the human mind);
- Relies on a set of techniques, including machine learning, to approximate a cognitive task; and,
- Is designed to act rationally by utilizing intelligent software or an embodied robot to achieve goals using perception, planning, reasoning, learning, communicating, decision-making, and action.
The HHS principles for use of trustworthy AI in government cover six areas: fair and impartial application; transparent and explainable data use; responsible and accountable governance and policies; robust and reliable outputs; safe and secure from risks; and, that privacy should be respected, data not used beyond its intended and stated use, and that it’s use is approved by the data owner or steward.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Assembly Appropriations Committee analysis, costs to the University of California (UC) of an unknown but potentially significant amount. According to UC Office of the President (UCOP), costs for vendor auditing will likely be passed on to UC health systems. UCOP also states the third-party audit requirement will disincentivize AI development within academic medical centers by adding substantial new cost for implementing internally-developed AI solutions. UCOP indicates these costs would reduce hospital revenue and create cost pressures for UC and the General Fund.
SUPPORT: (Verified 8/24/26)
America’s Physician Groups
California Association of Health Plans
California Medical Association
California Radiological Society
Kaiser Permanente
OCHIN Inc.
University of California
OPPOSITION: (Verified 8/24/26)
Advanced Medical Technology Association
ARGUMENTS IN SUPPORT: A coalition representing physicians, hospitals, physician groups, health plans, and other stakeholders, including the California Medical Association, Kaiser Permanente, and the California Association of Health Plans, submitted a letter of support to the most recent version of this bill. These supporters state that this bill would provide valuable information to health care deployers such as clinics, rural hospitals, independent physician practices, and others, before making significant technology/AI investments. Under this bill, developers would be required to provide information that will significantly help small health care providers to make informed decisions and better understand the strengths and limitations of tools they are considering using. This bill would equip deployers with necessary information to mitigate bias and risks of such tools and systems. These supporters state that AI holds extraordinary potential to transform and improve every aspect of health care, and this bill helps to level the playing field by ensuring all purchasers, regardless of size, have access to basic information necessary to determine whether a tool is appropriate for their patients and clinical workflows.
ARGUMENTS IN OPPOSITION: The Advanced Medical Technology Association (AdvaMed) submitted a letter of opposition to the most recent version of this bill. According to AdvaMed, unlike many other industries, the use of AI in medical technology is already subject to robust oversight and regulation by the FDA throughout the device’s lifecycle. FDA’s premarket review of AI-enabled devices includes an assessment of clinical relevant information, potential risk, the mitigation of unwanted bias, and performance validation on the device’s intended use and patient populations. Additionally, medical device manufacturers must follow strict federal requirements and regulations once devices are on the market, including monitoring and reporting of device malfunctions and serious adverse events. AdvaMed states that while they value the goal of this bill, they are concerned with the potential unintended consequences. AdvaMed’s concerns include the lack of clear definitions; that it overlooks the existing regulatory framework; and that it would establish duplicative California-specific requirements, and risks to confidentiality and intellectual property.
Prepared by: Vincent Marchand / Health / (916) 651-4111
8/24/26 17:09:33
SENATE THIRD READING
STR Bill Id:SB 503
Author:(Weber Pierson)
As Amended Ver:August 19, 2026
Majority vote
SUMMARY
Requires developers and deployers of artificial intelligence (AI) systems to make reasonable efforts to identify "clinical decision support systems," that may be at risk for bias, as defined, that are known or have a reasonably foreseeable risk of biased impacts when deployed in health care applications. Requires developers to disclose certain information to deployers of such systems. Allows developers to comply with this requirement by using a nationally recognized or widely adopted industry standard framework for such disclosure, or by sharing results of standardized assessments. Requires deployers to regularly monitor such systems and take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts.
Major Provisions
COMMENTS
AI in Health Care. AI is the mimicking of human intelligence by artificial systems. AI uses algorithms– sets of rules– to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI in health care is not new; AI models of varying degrees of sophistication have been developed and deployed in the health care setting for decades, and the use of AI is growing. According to the National Academies of Medicine, advancements in AI have the potential to transform health and medicine as we know it, but the potential for both breakthrough innovation and unintended consequences demands careful consideration.
Informational Hearing. On May 28, 2025, the Assembly Committees on Health and Privacy & Consumer Protection held an informational hearing titled, "Generative Artificial Intelligence in Health Care: Opportunities, Challenges, and Policy Implications." The hearing examined the history of AI in health care; current applications; a range of challenges that pose barriers to responsible, effective adoption of AI, including bias and other challenges; other considerations, such as workforce, cost, reimbursement, liability, and data privacy; and existing regulatory frameworks and best practices, including federal and state law and regulation and private efforts.
Current Use Cases. Researchers, health care providers and facilities, health plans and others are deploying AI for a range of tasks in biomedical and health research, as well as various administrative and clinical use cases.
Racial, Ethnic and Gender Bias in AI. There is a famous saying in computer science: "garbage in, garbage out." The performance of an AI is directly impacted by the quality, quantity, and relevance of the data used to train it. If the data used to train the AI is biased, the tool's outputs will be similarly biased and the results can be inaccurate when applied to populations not reflected in the training data. When automated decision systems are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. Numerous examples of bias have been described in the literature.
Affordable Care Act (ACA) Anti-Discrimination Regulations. Section 1557, the civil rights provision of the ACA, prohibits discrimination on the grounds of race, color, national origin, sex, age, or disability in certain health programs and activities. Section 1557(c) of the ACA authorizes the Health and Human Services (HHS) Secretary to promulgate regulations to implement the nondiscrimination requirements of Section 1557. Section 1557 only applies to "covered entities," that is, health programs and activities that receive federal financial assistance from HHS. Examples of types of covered entities under Section 1557 include hospitals, health clinics, physicians' practices, community health centers, nursing homes, rehabilitation centers, health insurance issuers, and state Medicaid agencies.
2024 HHS Office of Civil Rights (OCR) regulations issued pursuant to Section 1557 prohibit a covered entity from discriminating on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools (automated or AI tools used to support clinical decision-making). The regulation requires a covered entity to make reasonable efforts to identify uses of patient care decision support tools that may promote bias and make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities.
Alignment with Federal Section 1557 Regulations. This bill is similar to federal Section 1557 regulations in that deployers of these tools are required to make "reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool's use in its health programs or activities," similar to the federal regulation. However, the bill is different than federal regulations in key ways. It is narrower than the federal regulation in the types of tools it covers, it applies to fewer health care entities than the federal regulation applies to but extends its requirements to developers, which are not generally covered under Section 1557 regulations, and it specifically requires monitoring, which is not explicitly addressed in regulations, and it defines "protected characteristic" as those laid out in state civil rights law, which is broader than the federal regulation, which prohibits discrimination based on "race, color, national origin, sex, age, or disability."
How is Bias in AI Systems Identified, Mitigated, and Monitored? There is widespread awareness that bias is a problem that needs attention from developers and deployers of AI, and there is ongoing work to develop ways to measure and address it. Identifying, monitoring, and mitigating for bias may require a deployer to be aware of the literature on what types of systems may be prone to bias, understand how a model's training data compares to their patient population, conduct sensitivity analyses to see how calibrating a model in different ways effects the outputs of the model, and make technical adjustments to a model. For instance, a clinic might have to calibrate or adjust the model in a specific way to ensure it works effectively for their particular patient population. As industry standards continue to develop to support such monitoring, it is possible that more of these this work to mitigate bias will be done on the front end in the development of AI solutions, and thus the work of addressing bias may become less burdensome on individual providers over time.
Private and Industry Efforts on Transparency, Disclosure, and Evaluation. Transparency about AI models is critical to identifying the potential for bias, and deployers often do not receive standardized information on model development that is needed to meaningfully identify and address bias. An August 2022 survey by the Office of California Attorney General (AG) Rob Bonta found hospitals are using decision-making tools to make judgments about patients across many contexts, they rely on the vendor's assessment that the tools they use are ethical and unbiased, and they lack insight into vendors' data modeling. To address this lack of transparency and improve the effective and appropriate deployment of AI technology in health care, the Coalition for Health AI (CHAI), a large national collaborative effort of health systems, public and private organizations, academia, patient advocacy groups, and AI experts, has released a draft template for an "applied model card" that is intended to be published by a developer and describe key information about health AI models, in a manner somewhat similar to a "Nutrition Facts" label. If the model card were widely adopted, a number of elements included therein would assist a deployer of an AI model in identifying, mitigating, and monitoring for bias.
According to the Author
This bill is a crucial step towards ensuring fairness in health care by addressing the racial biases embedded in AI models and systems. This technology is becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. The author states that this bill will require collaboration between developers and healthcare facilities to identify AI tools used in the delivery of patient care and proactively work towards meaningfully reducing bias. By requiring identification, mitigation, and oversight, the author notes this bill will help promote safety, equity, and exceptional performance while protecting patients against avoidable harm. This bill was inspired by the 2023 California Reparations Task Force Report.
Arguments in Support
A coalition representing physicians, hospitals and health systems, physician groups, health plans, and other health care stakeholders supports this bill, stating it would provide a
level of transparency that will significantly and especially help small health care providers/deployers make informed decisions and better understand the strengths and limitations of tools they are considering using. Oakland Privacy, a citizen's coalition that works regionally to defend the right to privacy and enhance public transparency, writes in support that it is sensible to place into California statute a corresponding version of federal regulations that they believe support California's goals and priorities. Oakland Privacy notes, given the interesting definitions of Diversity Equity am Inclusion, or "DEI," and "discrimination" being promulgated by some federal agencies, it makes even more sense to capture the intent of federal regulations before they are twisted into shapes quite unlike their original intent.
Arguments in Opposition
AdvaMed opposes this bill, citing inadequate attention to existing regulatory frameworks such as the United States Food and Drug Administration (FDA) approval. Some organizations opposed a prior version of the bill, which required developers to submit their systems to third-party audits to demonstrate compliance. This audit requirement has been removed from the bill.
FISCAL COMMENTS
According to the Assembly Committee on Appropriations:
- The California Department of Public Health (CDPH) estimates costs of approximately $9.3 million in fiscal year (FY) 2026-27 and FY 2027-28 and $9.1 million in FY 2028-29 to develop regulations and ensure compliance (Licensing and Certification Fund).
- Costs to the University of California (UC) of an unknown but potentially significant amount. According to UC Office of the President (UCOP), costs for vendor auditing will likely be passed on to UC health systems. UCOP also states the third-party audit requirement will disincentivize AI development within academic medical centers by adding substantial new cost for implementing internally-developed AI solutions. UCOP indicates these costs would reduce hospital revenue and create cost pressures for UC and the General Fund.
- CDPH states it must (1) develop regulations defining "reasonable efforts" to mitigate risk for biased impacts in the AI system's outputs and (2) survey health facilities to ensure the facilities are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and to mitigate the risk for biased impacts in the system's outputs. CDPH estimates it will require 47 to 47.5 full-time equivalent (FTE) positions in FY 2026-27 through FY 2028-29, and 46.5 ongoing FTE positions starting FY 2029-30 to survey health facilities to ensure they are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and making reasonable efforts to mitigate the risk for biased impacts in the AI system's outputs.
VOTES
SENATE FLOOR: 38-0-2
YES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Cervantes, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
ABS, ABST OR NV: Limón, Reyes
ASM HEALTH: 16-0-0
YES: Bonta, Chen, Addis, Aguiar-Curry, Caloza, Rogers, Flora, Mark González, Elhawary, Patel, Ellis, Celeste Rodriguez, Sanchez, Schiavo, Sharp-Collins, Stefani
ASM PRIVACY AND CONSUMER PROTECTION: 13-0-2
YES: Bauer-Kahan, Dixon, Bryan, Irwin, Lowenthal, McKinnor, Ortega, Patterson, Pellerin, Petrie-Norris, Ward, Wicks, Wilson
ABS, ABST OR NV: DeMaio, Macedo
ASM APPROPRIATIONS: 15-0-0
YES: Wicks, Sanchez, Arambula, Calderon, Caloza, Dixon, Elhawary, Fong, Mark González, Ahrens, Pacheco, Pellerin, Solache, Ta, Tangipa
UPDATED
VERSION: August 19, 2026
CONSULTANT: Lisa Murawski / HEALTH / (916) 319-2097 FN: 0003991
SENATE THIRD READING
STR Bill Id:SB 503
Author:(Weber Pierson)
As Amended Ver:September 4, 2025
Majority vote
SUMMARY
Requires developers and deployers of artificial intelligence (AI) systems in specified health care applications to take steps to identify, mitigate, and monitor biased impacts. Beginning in 2027, requires developers and deployers to provide the California Department of Public Health (DPH) with a report identifying their efforts to comply with the bill's requirements and maintain documentation of their compliance efforts. Specifies DPH is not required to independently inspect, test, or evaluate the functionality of an AI system.
Major Provisions
COMMENTS
AI in Health Care. AI is the mimicking of human intelligence by artificial systems. AI uses algorithms– sets of rules– to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI in health care is not new; AI models of varying degrees of sophistication have been developed and deployed in the health care setting for decades, and the use of AI is growing. According to the National Academies of Medicine, advancements in AI have the potential to transform health and medicine as we know it, but the potential for both breakthrough innovation and unintended consequences demands careful consideration.
Informational Hearing. On May 28, 2025, the Assembly Committees on Health and Privacy & Consumer Protection held an informational hearing titled, "Generative Artificial Intelligence in Health Care: Opportunities, Challenges, and Policy Implications." The hearing examined the history of AI in health care; current applications; a range of challenges that pose barriers to responsible, effective adoption of AI, including bias and other challenges; other considerations, such as workforce, cost, reimbursement, liability, and data privacy; and existing regulatory frameworks and best practices, including federal and state law and regulation and private efforts.
Current Use Cases. Researchers, health care providers and facilities, health plans and others are deploying AI for a range of tasks in biomedical and health research, as well as various administrative and clinical use cases.
Racial, Ethnic and Gender Bias in AI. There is a famous saying in computer science: "garbage in, garbage out." The performance of an AI is directly impacted by the quality, quantity, and relevance of the data used to train it. If the data used to train the AI is biased, the tool's outputs will be similarly biased and the results can be inaccurate when applied to populations not reflected in the training data. When automated decision systems are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. Numerous examples of bias have been described in the literature.
Affordable Care Act (ACA) Anti-Discrimination Regulations. Section 1557, the civil rights provision of the ACA, prohibits discrimination on the grounds of race, color, national origin, sex, age, or disability in certain health programs and activities. Section 1557(c) of the ACA authorizes the Health and Human Services (HHS) Secretary to promulgate regulations to implement the nondiscrimination requirements of Section 1557. Section 1557 only applies to "covered entities," that is, health programs and activities that receive federal financial assistance from HHS. Examples of types of covered entities under Section 1557 include hospitals, health clinics, physicians' practices, community health centers, nursing homes, rehabilitation centers, health insurance issuers, and state Medicaid agencies.
2024 HHS Office of Civil Rights (OCR) regulations issued pursuant to Section 1557 prohibit a covered entity from discriminating on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools (automated or AI tools used to support clinical decision-making). The regulation requires a covered entity to make reasonable efforts to identify uses of patient care decision support tools that may promote bias and make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities.
Alignment with Federal Section 1557 Regulations. This bill is similar to federal Section 1557 regulations in that deployers of these tools are required to make "reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool's use in its health programs or activities," similar to the federal regulation. However, the bill is different than federal regulations in key ways. It is narrower than the federal regulation in the types of tools it covers, it applies to fewer health care entities than the federal regulation applies to but extends its requirements to developers, which are not generally covered under Section 1557 regulations, and it specifically requires monitoring, which is not explicitly addressed in regulations, and it defines "protected characteristic" as those laid out in state civil rights law, which is broader than the federal regulation, which prohibits discrimination based on "race, color, national origin, sex, age, or disability." It also requires developers to submit to independent third-party audits.
How is Bias in AI Systems Identified, Mitigated, and Monitored? There is widespread awareness that bias is a problem that needs attention from developers and deployers of AI, and there is ongoing work to develop ways to measure and address it. Identifying, monitoring, and mitigating for bias may require a deployer to be aware of the literature on what types of systems may be prone to bias, understand how a model's training data compares to their patient population, conduct sensitivity analyses to see how calibrating a model in different ways effects the outputs of the model, and make technical adjustments to a model. For instance, a clinic might have to calibrate or adjust the model in a specific way to ensure it works effectively for their particular patient population. As industry standards continue to develop to support such monitoring, it is possible that more of these this work to mitigate bias will be done on the front end in the development of AI solutions, and thus the work of addressing bias may become less burdensome on individual providers over time.
Private and Industry Efforts on Transparency, Disclosure, and Evaluation. Transparency about AI models is critical to identifying the potential for bias, and deployers often do not receive standardized information on model development that is needed to meaningfully identify and address bias. An August 2022 survey by the Office of California Attorney General (AG) Rob Bonta found hospitals are using decision-making tools to make judgments about patients across many contexts, they rely on the vendor's assessment that the tools they use are ethical and unbiased, and they lack insight into vendors' data modeling. To address this lack of transparency and improve the effective and appropriate deployment of AI technology in health care, the Coalition for Health AI (CHAI), a large national collaborative effort of health systems, public and private organizations, academia, patient advocacy groups, and AI experts, has released a draft template for an "applied model card" that is intended to be published by a developer and describe key information about health AI models, in a manner somewhat similar to a "Nutrition Facts" label. If the model card were widely adopted, a number of elements included therein would assist a deployer of an AI model in identifying, mitigating, and monitoring for bias.
Enforcement. DPH licenses and oversees clinics and health facilities. Other deployers to whom this bill applies, including physician's offices and offices of a group practice, are operated under the auspices of physician licensure and are not specifically regulated in California. Developers, similarly, are not subject to any specific state oversight regime that would monitor compliance. This bill, however, requires developers to report compliance to DPH.
According to the Author
This bill is a crucial step towards ensuring fairness in health care by addressing the racial biases embedded in AI models and systems. This technology is becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. The author states that this bill will require collaboration between developers and healthcare facilities to identify AI tools used in the delivery of patient care and proactively work towards meaningfully reducing bias. By requiring identification, mitigation, and oversight, the author notes this bill will help promote safety, equity, and exceptional performance while protecting patients against avoidable harm. This bill was inspired by the 2023 California Reparations Task Force Report.
Arguments in Support
Oakland Privacy, a citizen's coalition that works regionally to defend the right to privacy and enhance public transparency, writes in support that it is sensible to place into California statute a corresponding version of federal regulations that they believe support California's goals and priorities. Oakland Privacy notes, given the interesting definitions of "DEI" and "discrimination" being promulgated by some federal agencies, it makes even more sense to capture the intent of federal regulations before they are twisted into shapes quite unlike their original intent. The California Medical Association also supports this bill.
Arguments in Opposition
TechNet argues in opposition that this bill defines a "developer" so broadly that it could encompass any company whose AI system might be used in a healthcare setting—regardless of whether the company designed or intended its system to support clinical decision-making or health-resource allocation. AdvaMed, Biocom, and California Life Sciences express similar concerns to several provisions of this bill, citing inadequate attention to existing regulatory frameworks. Several organization opposed a prior version of the bill, which required developers to submit their systems to third-party audits to demonstrate compliance. This audit requirement has been removed from the bill.
FISCAL COMMENTS
According to the Assembly Committee on Appropriations:
- The California Department of Public Health (CDPH) estimates costs of approximately $9.3 million in fiscal year (FY) 2026-27 and FY 2027-28 and $9.1 million in FY 2028-29 to develop regulations and ensure compliance (Licensing and Certification Fund).
- Costs to the University of California (UC) of an unknown but potentially significant amount. According to UC Office of the President (UCOP), costs for vendor auditing will likely be passed on to UC health systems. UCOP also states the third-party audit requirement will disincentivize AI development within academic medical centers by adding substantial new cost for implementing internally-developed AI solutions. UCOP indicates these costs would reduce hospital revenue and create cost pressures for UC and the General Fund.
- CDPH states it must (1) develop regulations defining "reasonable efforts" to mitigate risk for biased impacts in the AI system's outputs and (2) survey health facilities to ensure the facilities are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and to mitigate the risk for biased impacts in the system's outputs. CDPH estimates it will require 47 to 47.5 full-time equivalent (FTE) positions in FY 2026-27 through FY 2028-29, and 46.5 ongoing FTE positions starting FY 2029-30 to survey health facilities to ensure they are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and making reasonable efforts to mitigate the risk for biased impacts in the AI system's outputs.
VOTES
SENATE FLOOR: 38-0-2
YES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Cervantes, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
ABS, ABST OR NV: Limón, Reyes
ASM HEALTH: 16-0-0
YES: Bonta, Chen, Addis, Aguiar-Curry, Caloza, Rogers, Flora, Mark González, Elhawary, Patel, Ellis, Celeste Rodriguez, Sanchez, Schiavo, Sharp-Collins, Stefani
ASM PRIVACY AND CONSUMER PROTECTION: 13-0-2
YES: Bauer-Kahan, Dixon, Bryan, Irwin, Lowenthal, McKinnor, Ortega, Patterson, Pellerin, Petrie-Norris, Ward, Wicks, Wilson
ABS, ABST OR NV: DeMaio, Macedo
ASM APPROPRIATIONS: 15-0-0
YES: Wicks, Sanchez, Arambula, Calderon, Caloza, Dixon, Elhawary, Fong, Mark González, Ahrens, Pacheco, Pellerin, Solache, Ta, Tangipa
UPDATED
VERSION: September 4, 2025
CONSULTANT: Lisa Murawski / HEALTH / (916) 319-2097 FN: 0001675
SENATE THIRD READING
STR Bill Id:SB 503
Author:(Weber Pierson)
As Amended Ver:July 17, 2025
Majority vote
SUMMARY
Requires developers and deployers of artificial intelligence (AI) systems in specified health care applications to take steps to identify, mitigate, and monitor biased impacts. Beginning in 2030, requires developers to use an independent third-party auditor to assess compliance and to make a high-level summary of the results of an audit publicly available.
Major Provisions
COMMENTS
AI in Health Care. AI is the mimicking of human intelligence by artificial systems. AI uses algorithms– sets of rules– to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI in health care is not new; AI models of varying degrees of sophistication have been developed and deployed in the health care setting for decades, and the use of AI is growing. According to the National Academies of Medicine, advancements in AI have the potential to transform health and medicine as we know it, but the potential for both breakthrough innovation and unintended consequences demands careful consideration.
Informational Hearing. On May 28, 2025, the Assembly Committees on Health and Privacy & Consumer Protection held an informational hearing titled, "Generative Artificial Intelligence in Health Care: Opportunities, Challenges, and Policy Implications." The hearing examined the history of AI in health care; current applications; a range of challenges that pose barriers to responsible, effective adoption of AI, including bias and other challenges; other considerations, such as workforce, cost, reimbursement, liability, and data privacy; and existing regulatory frameworks and best practices, including federal and state law and regulation and private efforts.
Current Use Cases. Researchers, health care providers and facilities, health plans and others are deploying AI for a range of tasks in biomedical and health research, as well as various administrative and clinical use cases.
Racial, Ethnic and Gender Bias in AI. There is a famous saying in computer science: "garbage in, garbage out." The performance of an AI is directly impacted by the quality, quantity, and relevance of the data used to train it. If the data used to train the AI is biased, the tool's outputs will be similarly biased and the results can be inaccurate when applied to populations not reflected in the training data. When automated decision systems are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. Numerous examples of bias have been described in the literature.
Affordable Care Act (ACA) Anti-Discrimination Regulations. Section 1557, the civil rights provision of the ACA, prohibits discrimination on the grounds of race, color, national origin, sex, age, or disability in certain health programs and activities. Section 1557(c) of the ACA authorizes the Health and Human Services (HHS) Secretary to promulgate regulations to implement the nondiscrimination requirements of Section 1557. Section 1557 only applies to "covered entities," that is, health programs and activities that receive federal financial assistance from HHS. Examples of types of covered entities under Section 1557 include hospitals, health clinics, physicians' practices, community health centers, nursing homes, rehabilitation centers, health insurance issuers, and state Medicaid agencies.
2024 HHS Office of Civil Rights (OCR) regulations issued pursuant to Section 1557 prohibit a covered entity from discriminating on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools (automated or AI tools used to support clinical decision-making). The regulation requires a covered entity to make reasonable efforts to identify uses of patient care decision support tools that may promote bias and make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities.
Alignment with Federal Section 1557 Regulations. This bill is similar to federal Section 1557 regulations in that deployers of these tools are required to make "reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool's use in its health programs or activities," similar to the federal regulation. However, the bill is different than federal regulations in key ways. It is narrower than the federal regulation in the types of tools it covers, it applies to fewer health care entities than the federal regulation applies to but extends its requirements to developers, which are not generally covered under Section 1557 regulations, and it specifically requires monitoring, which is not explicitly addressed in regulations, and it defines "protected characteristic" as those laid out in state civil rights law, which is broader than the federal regulation, which prohibits discrimination based on "race, color, national origin, sex, age, or disability." It also requires developers to submit to independent third-party audits.
How is Bias in AI Systems Identified, Mitigated, and Monitored? There is widespread awareness that bias is a problem that needs attention from developers and deployers of AI, and there is ongoing work to develop ways to measure and address it. Identifying, monitoring, and mitigating for bias may require a deployer to be aware of the literature on what types of systems may be prone to bias, understand how a model's training data compares to their patient population, conduct sensitivity analyses to see how calibrating a model in different ways effects the outputs of the model, and make technical adjustments to a model. For instance, a clinic might have to calibrate or adjust the model in a specific way to ensure it works effectively for their particular patient population. As industry standards continue to develop to support such monitoring, it is possible that more of these this work to mitigate bias will be done on the front end in the development of AI solutions, and thus the work of addressing bias may become less burdensome on individual providers over time.
Private and Industry Efforts on Transparency, Disclosure, and Evaluation. Transparency about AI models is critical to identifying the potential for bias, and deployers often do not receive standardized information on model development that is needed to meaningfully identify and address bias. An August 2022 survey by the Office of California Attorney General (AG) Rob Bonta found hospitals are using decision-making tools to make judgments about patients across many contexts, they rely on the vendor's assessment that the tools they use are ethical and unbiased, and they lack insight into vendors' data modeling. To address this lack of transparency and improve the effective and appropriate deployment of AI technology in health care, the Coalition for Health AI (CHAI), a large national collaborative effort of health systems, public and private organizations, academia, patient advocacy groups, and AI experts, has released a draft template for an "applied model card" that is intended to be published by a developer and describe key information about health AI models, in a manner somewhat similar to a "Nutrition Facts" label. If the model card were widely adopted, a number of elements included therein would assist a deployer of an AI model in identifying, mitigating, and monitoring for bias.
Enforcement. The Department of Public Health (DPH) licenses and oversees clinics and health facilities. Other deployers to whom this bill applies, including physician's offices and offices of a group practice, are operated under the auspices of physician licensure and are not specifically regulated in California. Developers, similarly, are not subject to any specific state oversight regime that would monitor compliance.
According to the Author
This bill is a crucial step towards ensuring fairness in health care by addressing the racial biases embedded in AI models and systems. This technology is becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. The author states that this bill will require collaboration between developers and healthcare facilities to identify AI tools used in the delivery of patient care and proactively work towards meaningfully reducing bias. By requiring identification, mitigation, and oversight, the author notes this bill will help promote safety, equity, and exceptional performance while protecting patients against avoidable harm. This bill was inspired by the 2023 California Reparations Task Force Report.
Arguments in Support
Oakland Privacy, a citizen's coalition that works regionally to defend the right to privacy and enhance public transparency, writes in support that it is sensible to place into California statute a corresponding version of federal regulations that they believe support California's goals and priorities. Oakland Privacy notes, given the interesting definitions of "DEI" and "discrimination" being promulgated by some federal agencies, it makes even more sense to capture the intent of federal regulations before they are twisted into shapes quite unlike their original intent. The California Medical Association also supports this bill.
Arguments in Opposition
According to TechNet, a national, bipartisan network of technology CEOs and senior
executives, this bill would add confusing new requirements and costly third-party audits that could delay or deter deployment of life-saving tools. TechNet argues the bill defines a "developer" so broadly that it could encompass any company whose AI system might be used in a healthcare setting—regardless of whether the company designed or intended its system to support clinical decision-making or health-resource allocation. In addition, TechNet opposes the mandate for third-party audits, asserting the AI audit ecosystem is still immature and lacks standards governing auditor qualifications, methods, or accountability. TechNet argues a more effective and less costly approach would be to require developers to publish system documentation describing bias evaluation and mitigation practices, so stakeholders and regulators can evaluate compliance. TechNet respectfully urges amendments to: (1) narrow the developer definition, (2) remove or significantly rework the third-party audit requirement, and (3) align the bill with existing legal frameworks and widely accepted standards.
AdvaMed, Biocom, and California Life Sciences express similar concerns regarding several provisions of this bill, citing a lack of clear definitions, confidentiality and intellectual property risks, inadequate attention to existing regulatory frameworks, a lack of established standards, and inadequate expertise among auditors to assess AI systems for bias mitigation. These organizations expressing opposition to the bill unless amended to account for the existing regulatory framework to avoid unintended consequences.
ATA Action, the American Telemedicine Association's affiliated trade association, raises similar concerns with the newly inserted audit requirement, which ATA Action argues will create confusion, lead to significant added costs, and potentially stifle innovation and investment in this growing field.
FISCAL COMMENTS
According to the Assembly Committee on Appropriations:
- The California Department of Public Health (CDPH) estimates costs of approximately $9.3 million in fiscal year (FY) 2026-27 and FY 2027-28 and $9.1 million in FY 2028-29 to develop regulations and ensure compliance (Licensing and Certification Fund).
- Costs to the University of California (UC) of an unknown but potentially significant amount. According to UC Office of the President (UCOP), costs for vendor auditing will likely be passed on to UC health systems. UCOP also states the third-party audit requirement will disincentivize AI development within academic medical centers by adding substantial new cost for implementing internally-developed AI solutions. UCOP indicates these costs would reduce hospital revenue and create cost pressures for UC and the General Fund.
- CDPH states it must (1) develop regulations defining "reasonable efforts" to mitigate risk for biased impacts in the AI system's outputs and (2) survey health facilities to ensure the facilities are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and to mitigate the risk for biased impacts in the system's outputs. CDPH estimates it will require 47 to 47.5 full-time equivalent (FTE) positions in FY 2026-27 through FY 2028-29, and 46.5 ongoing FTE positions starting FY 2029-30 to survey health facilities to ensure they are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and making reasonable efforts to mitigate the risk for biased impacts in the AI system's outputs.
VOTES
SENATE FLOOR: 38-0-2
YES: Allen, Alvarado-Gil, Archuleta, Arreguín, Ashby, Becker, Blakespear, Cabaldon, Caballero, Cervantes, Choi, Cortese, Dahle, Durazo, Gonzalez, Grayson, Grove, Hurtado, Jones, Laird, McGuire, McNerney, Menjivar, Niello, Ochoa Bogh, Padilla, Pérez, Richardson, Rubio, Seyarto, Smallwood-Cuevas, Stern, Strickland, Umberg, Valladares, Wahab, Weber Pierson, Wiener
ABS, ABST OR NV: Limón, Reyes
ASM HEALTH: 16-0-0
YES: Bonta, Chen, Addis, Aguiar-Curry, Caloza, Rogers, Flora, Mark González, Elhawary, Patel, Ellis, Celeste Rodriguez, Sanchez, Schiavo, Sharp-Collins, Stefani
ASM PRIVACY AND CONSUMER PROTECTION: 13-0-2
YES: Bauer-Kahan, Dixon, Bryan, Irwin, Lowenthal, McKinnor, Ortega, Patterson, Pellerin, Petrie-Norris, Ward, Wicks, Wilson
ABS, ABST OR NV: DeMaio, Macedo
ASM APPROPRIATIONS: 15-0-0
YES: Wicks, Sanchez, Arambula, Calderon, Caloza, Dixon, Elhawary, Fong, Mark González, Ahrens, Pacheco, Pellerin, Solache, Ta, Tangipa
UPDATED
VERSION: July 17, 2025
CONSULTANT: Lisa Murawski / HEALTH / (916) 319-2097 FN: 0001413
Date of Hearing: August 20, 2025
ASSEMBLY COMMITTEE ON APPROPRIATIONS
Buffy Wicks, Chair
SBPCA Bill Id:SB 503
Author:(Weber Pierson) – As Amended Ver:July 17, 2025
Policy Committee: | Health | Vote: | 16 - 0 |
Privacy and Consumer Protection | 13 - 0 | ||
|
|
Urgency: No State Mandated Local Program: No Reimbursable: No
SUMMARY:
This bill requires developers and deployers of artificial intelligence (AI) systems identify AI systems used in certain health care applications that may risk biased impacts, and make reasonable efforts to mitigate risk of bias. The bill requires use of an independent third-party auditor to assess compliance with the duties to mitigate risks of bias.
Specifically, this bill:
- Specifies developers and deployers of AI systems have an ongoing duty to make reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the system’s outputs resulting from use of the system in health programs or activities.
- Requires developers and deployers make reasonable efforts to mitigate the risk for biased impacts in the system’s outputs resulting from use of the AI systems in item 1, above, in health programs or activities.
- Requires deployers regularly monitor AI systems in item 1, above, and take reasonable and proportionate steps to mitigate any bias that may occur.
- Provides that, for purposes of this bill, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
- Requires, beginning January 1, 2030, and at least annually thereafter, a developer submit their AI systems to an independent third-party auditor to assess whether the developer has complied with their duties pursuant to items 1 through 3, above.
- Requires a developer subject to item 5, above, make a high-level summary of the results of the required audit publicly available, at no cost, on the developer’s internet website.
- Defines the following terms, among others:
- “Biased impact” means an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
- “Deployer” means a person, partnership, state or local governmental agency, corporation, or developer that uses an AI system to support clinical decision-making or health care resource allocation.
- “Developer” means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an AI system for commercial or public use to support clinical decision-making or health care resource allocation.
- “AI” is defined by reference to a provision in the Government Code.
- “Protected characteristic” means a characteristic listed in the Unruh Civil Rights Act.
- Provides that this bill does not supplant or replace any other applicable provision of state law regulating the use of AI or automated decision systems, and compliance with this section may not be used as a defense to a claim of unlawful discrimination.
FISCAL EFFECT:
The California Department of Public Health (CDPH) estimates costs of approximately $9.3 million in fiscal year (FY) 2026-27 and FY 2027-28 and $9.1 million in FY 2028-29 to develop regulations and ensure compliance (Licensing and Certification Fund).
Costs to the University of California (UC) of an unknown but potentially significant amount. According to UC Office of the President (UCOP), costs for vendor auditing will likely be passed on to UC health systems. UCOP also states the third-party audit requirement will disincentivize AI development within academic medical centers by adding substantial new cost for implementing internally-developed AI solutions. UCOP indicates these costs would reduce hospital revenue and create cost pressures for UC and the General Fund.
CDPH states it must (1) develop regulations defining “reasonable efforts” to mitigate risk for biased impacts in the AI system’s outputs and (2) survey health facilities to ensure the facilities are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and to mitigate the risk for biased impacts in the system’s outputs. CDPH estimates it will require 47 to 47.5 full-time equivalent (FTE) positions in FY 2026-27 through FY 2028-29, and 46.5 ongoing FTE positions starting FY 2029-30 to survey health facilities to ensure they are making reasonable efforts to identify AI systems used to support clinical decision-making or health care resource allocation and making reasonable efforts to mitigate the risk for biased impacts in the AI system’s outputs.
COMMENTS:
- Purpose. According to the author:
[This bill] is a crucial step towards ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. This technology is becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill will require collaboration between developers and healthcare facilities to identify AI tools used in the delivery of patient care and proactively work towards meaningfully reducing bias. By requiring identification, mitigation, and oversight, [this bill] will help promote safety, equity, and exceptional performance while protecting patients against avoidable harm.
- Background. As noted in the background paper for the Assembly Health Committee and Assembly Privacy and Consumer Protection Committee Joint Informational Hearing on Generative AI (GenAI) in health care, AI algorithms, machine learning, and predictive AI models of varying degrees of sophistication have been developed and deployed for years. After recent advances, particularly in natural language processing, interest in and use of AI health care applications have proliferated. AI systems are often trained on historical datasets, which may underrepresent certain demographic groups and frequently overrepresent white individuals, who have historically accessed healthcare at higher rates than people of color. As a result, the outputs of such systems may perpetuate or exacerbate existing health disparities. For example, an AI tool developed by a health care company and used by providers across the country to offer care management services was found to assign Black patients lower likelihoods of adverse health outcomes than equally at-risk white patients. The authors determined the bias resulted from the tool having been designed to predict health care costs instead of needs: because the health care system has historically spent less on care for Black patients than white patients for the same health conditions, the tool was making a prediction that mirrored and perpetuated past discrimination. Another example showed bias in double-booking appointments for patients from marginalized populations.
- Stakeholder Concerns. The California Hospital Association (CHA) supports this bill if amended to remove the requirement for use of an independent third-party auditor. CHA contends these companies do not exist in health care. CHA expresses concerns about potentially very high costs to developers for such audits, and notes the high costs would be passed on to deployers.
Analysis Prepared by: Allegra Kim / APPR. / (916) 319-2081
Date of Hearing: July 16, 2025
Fiscal:
ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION
Rebecca Bauer-Kahan, Chair
SBPCA Bill Id:SB 503
Author:(Weber Pierson) – As Amended Ver:July 10, 2025
SENATE VOTE: 38-0
PROPOSED AMENDMENTS
SUBJECT: Health care services: artificial intelligence
SYNOPSIS
In 2020, the Legislature enacted AB 3121 (Weber, Ch. 319, Stats. 2020), which established the Task Force to Study and Develop Reparation Proposals for African Americans, with Special Consideration for African Americans Who Are Descendants of Persons Enslaved in the United States. In 2023, the Task Force released a report detailing both the historical injustices of slavery and their continuing influence in contemporary society. The report also offered a series of recommendations for addressing those enduring harms.
With respect to healthcare, the Task Force observed: “Racial disparities in African American health outcomes occur today as a culmination of historical racial inequality, discriminatory health policy, and persistent racial discrimination in many sectors of life in the United States.” As technology advances, these disparities can manifest in new ways, including the underrepresentation of people of color, particularly Black Californians, in the data used to train algorithms and artificial intelligence (AI) systems. This underrepresentation may contribute to disparate outcomes, such as misdiagnoses, denial or degradation of care, and neglect by systems intended to promote health and wellbeing.
This bill, sponsored by the California Black Health Network, seeks to address these historical and structural inequities by imposing requirements on developers and deployers of AI systems used in healthcare settings. Specifically, it requires developers and deployers to identify AI systems they use to support clinical decisionmaking or resource allocation that are known to, or may reasonably be foreseen to, pose a risk of adverse impact on a protected class. Additionally, the bill imposes a duty on both developers and deployers to take reasonable measures to mitigate risks of biased outputs. Deployers must also monitor the AI systems under their control and take reasonable corrective action if biased impacts are detected.
This measure is supported by Kaiser Permanente, the California Medical Association, and the California Hospital Association. The bill has no registered opposition. It passed the Health Committee on a 16-0 vote.
Committee amendments, outlined in Comment #6, would enhance accountability for developers of these technologies by requiring independent third-party audits to assess compliance with this bill beginning in 2030.
THIS BILL:
- Requires developers of artificial intelligence systems, and health facilities, clinics, physician’s offices, or offices of a group practice, to have an ongoing duty to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the system’s outputs resulting from use of the system in health programs or activities.
- Requires developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the system’s outputs resulting from use of the system in health programs or activities.
- Requires deployers shall regularly monitor these artificial intelligence systems and take reasonable and proportionate steps to mitigate any bias that may occur.
- Defines the terms as follows:
- “Biased impact” to mean an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
- “Deployer” to mean a person, partnership, state or local governmental agency, corporation, or developer that uses an artificial intelligence system to support clinical decisionmaking or health care resource allocation
- “Developer” to mean a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an artificial intelligence system for commercial or public use to support clinical decisionmaking and or health care resource allocation.
EXISTING LAW:
- Prohibits, pursuant to the ACA, an individual, on the ground prohibited under title VI of the Civil Rights Act of 1964, title IX of the Education Amendments of 1972, the Age Discrimination Act of 1975, or section 504 of the Rehabilitation Act of 1973, be excluded from participation in, be denied the benefits of, or be subjected to discrimination under, any health program or activity, any part of which is receiving Federal financial assistance, including credits, subsidies, or contracts of insurance, or under any program or activity that is administered by an Executive Agency or other entity, as provided. (42 U.S.C. § 18116 (“Section 1557”).)
- Provides that the enforcement mechanisms provided for and available under such title VI, title IX, section 504, and Age Discrimination Act shall apply for purposes of violations of the above. Authorizes the Secretary to promulgate regulations relevant thereto. (42 U.S.C. § 18116.)
- Provides the following regulatory guidelines with regard to the above:
- A covered entity must not discriminate on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools.
- A covered entity has an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability.
- For each patient care decision support tool identified in (b), a covered entity must make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities. (45 C.F.R. § 92.210.)
- Establishes the Unruh Civil Rights Act (“Unruh”), which provides that all persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever. (Civ. Code § 51.)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk automated decision systems (ADS) that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms:
- “Automated decision system” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. “Automated decision system” does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
- “High-risk automated decision system” means an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. (Gov. Code § 11546.45.5.)
COMMENTS:
1) Author’s statement. According to the author:
SB 503 is a crucial step towards ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. This technology is becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill will require collaboration between developers and healthcare facilities to identify AI tools used in the delivery of patient care and proactively work towards meaningfully reducing bias. By requiring identification, mitigation, and oversight, SB 503 will help promote safety, equity, and exceptional performance while protecting patients against avoidable harm.
2) California Reparation Report and Healthcare. In 2020, the Legislature enacted AB 3121 (Weber, Ch. 319, Stats. 2020), which established the Task Force to Study and Develop Reparation Proposals for African Americans, with special consideration for those who are descendants of individuals enslaved in the United States. The Task Force’s work culminated in the publication of The California Reparations Report in 2023, which documents the ongoing and compounding harms experienced by African Americans as a result of slavery and its enduring legacy in contemporary society. With respect to healthcare, the report states:
Centuries of exposure to racism has contributed to a serious decline in African American physical and mental health. African Americans die at disproportionately higher rates from preventable health problems. Doctors are more likely to misdiagnose African Americans, leading to disparate outcomes in mental health. African American women face high rates of maternal death and adverse birth outcomes—even Black women with the highest education attainment have the worst birth outcomes across all women in America. African American children face poverty, malnutrition, and worse health than that of white American children. The mismanagement of public health crises by county, state, and federal governments has resulted in an undue burden of disease and death in African American communities—particularly during the COVID-19 pandemic. Despite this, in the face of overwhelming oppression, African American healthcare providers, patients, and community members, nonetheless, have worked to build healthy communities and fight for a more equitable healthcare system.[1]
The report further documents the historical injustices within California’s healthcare system, which has systematically and consistently disadvantaged Black Californians. It includes a section highlighting how advances in technology have often failed to benefit Black communities, resulting in continued disparities in health outcomes and access to healthcare services:
Algorithms are widely used in U.S. hospitals to refer people to health programs that improve a patient’s care— however, at least one widely-used algorithm was found to systematically discriminate against Black patients. This algorithm led to African American patients receiving less referrals for programs that provided personalized care—despite being just as sick as white patients.
African Americans are less likely to be treated for skin diseases due to the lack of medical research and training for diagnosing skin conditions for those with darker skin. Most medical textbooks and journals that assist dermatologists in diagnosing skin disorders do not include images of skin conditions as they appear on African Americans. Images of darker skin with skin conditions caused by COVID-19, skin cancer, psoriasis, rosacea, and melanoma often do not appear in medical textbooks and journals. Doctors routinely miss these diagnoses for African American patients because they are not trained to identify or treat skin conditions for African American patients. Consequently, discriminatory medical research and technology has resulted in worsening health disparities that harm African Americans.[2]
3) AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.
In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.
AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.
4) AI in Healthcare. As noted in the background paper for the Joint hearing between this Committee and the Assembly Health Committee, AI in healthcare is not a new phenomenon:
AI in health care is not new; AI algorithms, machine learning, and predictive AI models of varying degrees of sophistication have been developed and deployed for years. Some of the first applications were developed in the 1970’s and 1980’s. INTERNIST-1, which used a search algorithm to arrive at clinical diagnoses based on patients’ symptoms, was created in 1971. ELIZA, a rules-based mental health therapy chatbot program, was developed even earlier. In 2007, IBM created the open-domain question-answering system, “Watson.” In 2011, Watson won first place on Jeopardy and, in 2017, neurologists used it to identify key proteins that are altered in patients with Amyotrophic lateral sclerosis (ALS). Later, scientists at GoogleDeepMind shared a 2024 Nobel Prize in Chemistry for developing an AI model called AlphaFold2 to predict a protein’s 3D structure from its amino-acid sequence, which is reportedly accelerating breakthroughs in biology and drug development.
With the recent advancement of GenAI, particularly in natural language processing, interest in, use of, and hype over AI has grown rapidly and health care applications have proliferated. According to the National Academy of Medicine (NAM), GenAI and large language models (models designed for natural language processing tasks, or LLMs) have the potential to transform health and medicine as we know it: improving health care delivery, advancing medical research, and augmenting the capacity of clinicians to provide personalized care at an unprecedented scale. However, NAM also notes that the potential for both breakthrough innovation and unintended consequences demands careful consideration.[3]
Nevertheless, the maturity of a technology does not guarantee that it functions as intended. AI systems are often trained on historical datasets, which may underrepresent certain demographic groups and contribute to disparate health outcomes. These datasets frequently overrepresent white individuals, who have historically accessed healthcare at higher rates than people of color. As a result, the outputs of such systems may perpetuate or exacerbate existing health disparities. The Assembly Health Committee Analysis outlines several examples of how these disparities have manifested in practice:
In their work on mitigating bias in AI, the Berkeley Haas Center for Equity, Gender and Leadership (Center) tracks publicly available instances of bias in AI systems using machine leaning. In their analysis of around 133 biased systems across industries from 1988 to the present day, the Center found that 44% (59 systems) demonstrate gender bias, with 26% (34 systems) exhibiting both gender and racial bias.
When automated decision systems are deployed in healthcare, biased historical data can lead to patients being recommended substandard care on the basis of their race or ethnicity. In 2007, an automated decision system was developed to help doctors estimate whether it was safe for people who had delivered previous children through cesarean section to deliver subsequent children vaginally– a procedure that carries some risk. The system considered relevant factors as it made its decision, such as the woman’s age, her reason for the previous cesarean, and how long ago the cesarean had been performed. However, a 2017 study found that the system was biased; it predicted Black and Latino people were less likely to have a successful vaginal birth after a cesarean than similar non-Hispanic white women. As a result, doctors performed more cesareans on Black and Latino people than on white people, perpetuating historical racial and ethnic biases.
Similarly, in 2019, a study discovered harmful racial bias in an AI tool developed by the health care company Optum – a subsidiary of UnitedHealth Group – and used by providers across the country to offer care management services. The tool assigned Black patients lower likelihoods of adverse health outcomes than equally at-risk white patients. The authors found that this happened because the tool was designed to predict healthcare costs instead of needs. Because the healthcare system has historically spent less on care for Black patients than white patients for the same health conditions, the tool was issuing a prediction that mirrored and perpetuated past discrimination.
The University of California San Francisco also reported bias in an algorithm used to identify potential appointment no-shows to facilitate double-booking for appointments. The program was confirmed to result in low-resourced and marginalized populations being double-booked more often than others, reflecting underlying structural inequalities and highlighting how these tools, if not studied and corrected for bias, that can create feedback loops that worsen discrimination.
5) Federal Section 1557 Regulations and what this bill would do. Section 1557, the civil rights provision of the Affordable Care Act (ACA), prohibits discrimination on the basis of race, color, national origin, sex, age, or disability in certain health programs and activities.[4] It authorizes the Secretary of Health and Human Services (HHS) to issue regulations to implement these nondiscrimination requirements, and in 2024, HHS promulgated the following regulations:
- A covered entity must not discriminate on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools.
- A covered entity has an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability.
- For each patient care decision support tool identified in ii) above, a covered entity must make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities.
Section 1557 applies only to “covered entities,” meaning health programs and activities that receive federal financial assistance from HHS. Examples of covered entities include hospitals, health clinics, physicians’ practices, community health centers, nursing homes, rehabilitation centers, health insurance issuers, and state Medicaid agencies. The Federal Register outlines the scope of the term “patient care decision support tools,” which includes: automated decision systems (ADS); artificial intelligence (AI); flowcharts; formulas; equations; calculators; algorithms; utilization management applications; software as medical devices (SaMDs); software in medical devices (SiMDs); screening, risk assessment, and eligibility tools; and diagnostic and treatment guidance tools.[5]
This bill codifies similar principles to those articulated in Section 1557 by imposing a duty to mitigate harms associated with AI systems, many of which are likely to fall within the categories described by the Federal Register and used in the healthcare sector. Notably, the bill extends obligations to developers, who are not covered under Section 1557, as well as to health facilities, clinics, physician offices, and group practices.
Under the bill, developers and deployers must exercise due diligence to identify AI systems used to support clinical decision-making or healthcare resource allocation that present a foreseeable risk of biased outcomes. The bill defines “biased impact” as an unintended adverse effect on healthcare access, quality, or outcomes based on a protected characteristic, as defined under the Unruh Civil Rights Act, a broader set of characteristics than those covered under Section 1557. In addition, the bill requires both developers and deployers to take reasonable steps to mitigate the risk of biased impacts. Deployers are also obligated to regularly monitor these AI systems and implement reasonable measures to address any bias that arises.
6) Amendments. A major challenge in imposing requirements on developers and deployers of AI systems is ensuring compliance and determining whether a standard of care is being upheld, particularly when many users of these technologies are not well-versed in relevant processes or standards. To address this concern, the Committee has adopted the use of third-party auditors as a guiding principle this session (as reflected in Bauer-Kahan AB 1018 and Wiener SB 53, also being heard today), with auditing requirements set to take effect in 2030. This delayed onset provides a four-year runway for the development of a market. Accordingly, the author has agreed to amendments that would adopt an auditing requirement for developers of these technologies that are used to support clinical decision-making or health care resource allocation which will also begin in 2030. The inclusion of this language would encompass deployers who also serve as developers. The intent is to capture only the largest and most well-resourced entities that are capable of complying with the auditing requirement.
However, due to timeline constraints, the current amended language will broadly apply to developers. In upcoming discussions with stakeholders, if it becomes apparent that the provision may inadvertently capture safety net providers or other resource-constrained health organizations, the Committee is committed to working with the author to adopt further amendments to ensure the intent is properly effectuated.
To be clear, the auditing requirement will not apply to entities that only deploy the technology. The author has also agreed to adopt various clarifying and technical amendments. The amendments are as follows:
SECTION 1. Section 1339.76 is added to the Health and Safety Code, to read:
1339.76. (a) (1) Developers and deployers of artificial intelligence systems, and health facilities, clinics, physician’s offices, or offices of a group practice, shall have an ongoing duty to make reasonable efforts to identify artificial intelligence systems used to support clinical decisionmaking or health care resource allocation that are known or have a reasonably foreseeable risk of biased impacts in the system’s outputs resulting from use of the system in health programs or activities.
(2) Developers and deployers shall make reasonable efforts to mitigate the risk for biased impacts in the system’s outputs resulting from use of the systems identified pursuant to paragraph (1) in health programs or activities.
(3) Deployers shall regularly monitor these artificial intelligence systems identified pursuant to paragraph (1) and take reasonable and proportionate steps to mitigate any bias that may occur.
(b) For purposes of this section, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
(c)(1) Except as provided in subparagraph (B), beginning January 1, 2030, and at least annually thereafter, a developer shall submit their artificial intelligence systems to an independent third-party auditor to assess whether the developer has complied with their duties pursuant to subdivision (a).
(2) An entity subject to paragraph (1) shall make a high-level summary of the results of an audit required by that paragraph publicly available at no cost to a person who accesses the developer’s internet website.
(c) For purposes of this section, the following definitions apply:
(1) “Biased impact” means an unintended adverse impact, including diminished access to health care, quality of care, or outcomes, on an individual based on their protected characteristics.
(2) “Clinic” has the same meaning as defined in Section 1200 or 1200.1.
(3) “Deployer” means a health facility, clinic, physician’s office, or office of a group practice person, partnership, state or local governmental agency, corporation, or developer that uses an artificial intelligence system to support clinical decisionmaking or health care resource allocation.
(4) “Developer” means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an artificial intelligence system for commercial or public use to support clinical decisionmaking or health care resource allocation.
(5) “Health care provider” means a person licensed or certified pursuant to Division 2 (commencing with Section 500) of the Business and Professions Code.
(6) “Health facility” has the same meaning as Section 1250.
(7) “Office of a group practice” means an office or offices in which two or more physicians are legally organized as a partnership, professional corporation, or nonprofit corporation licensed according to subdivision (a) of Section 1204.
(8) “Artificial intelligence” has the same meaning as in Section 11546.45.5 of the Government Code.
(9) “Physician’s office” means an office of a physician in solo practice.
(10) “Protected characteristic” means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
(d) This section is in addition to and does not supplant or replace any other applicable provision of state law regulating the use of artificial intelligence or automated decision systems. Compliance with this section shall not be used as a defense to a claim of unlawful discrimination.
ARGUMENTS IN SUPPORT: Kaiser Permanente, writes in support:
Kaiser Permanente is pleased to support SB 503 (Weber-Pierson), which as amended June 30th, would require artificial intelligence (AI) developers and deployers to make reasonable efforts to mitigate biased impacts when AI tools are used in health programs and activities. As amended, the bill would also require deployers to regularly monitor such AI systems and tools to take steps in mitigating potential bias outcomes.
As one of the nation’s largest integrated health care organizations with a long history of adopting new technologies and applying them broadly, we are leading the responsible use of artificial intelligence across the health care sector. We use AI-based tools to improve health outcomes and the member care experience. And we use them without compromising patient safety, privacy, or the quality of our care. AI never makes medical decisions — our physicians and care teams do.
Artificial intelligence (AI) holds extraordinary potential to transform and improve every aspect of health care by enhancing care delivery and high quality, patient experiences, health outcomes, clinical safety, and affordability. AI has played a role in health care for decades and has improved overall health outcomes. Artificial intelligence has also played a significant role in supporting a health care field that adheres to high standards of patient safety, privacy, responsible use, and ethical decision-making. These same principles must guide AI’s continued development and deployment to ensure it improves patient care while maintaining trust.
Kaiser Permanente believes SB 503 is a measured and practical first approach in ensuring AI systems and tools are safe, secure, and reliable. AI must be trustworthy and for patients and the larger community to trust AI, we must do our best to prevent negative outcomes and maximize positive outcomes that benefit people. SB 503 is crucial for building trust in AI.
REGISTERED SUPPORT / OPPOSITION:
Support
California Hospital Association
California Medical Association (CMA)
Kaiser Permanente
Oakland Privacy
Opposition
None on file.
Analysis Prepared by: John Bennett / P. & C.P. / (916) 319-2200
Task Force to Study and Develop Reparation Proposals for African Americans, “Chapter 12 - Mental and Physical Harm and Neglect”, Department of Justice (June 29, 2023), https://oag.ca.gov/ab3121/report. ↑
Ibid. ↑
Joint Informational Hearing Assembly Health and Privacy & Consumer Protection Committees Generative Artificial Intelligence in Health Care background paper can be found at https://ahea.assembly.ca.gov/hearings/2025-2026-informational-hearings. ↑
Details regarding Section 1557 of the ACA can be found at https://www.hhs.gov/civil-rights/for-individuals/section-1557/fs-sex-discrimination/index.html. ↑
89 Fed. Reg. 37522 ↑
SENATE RULES COMMITTEE Office of Senate Floor Analyses (916) 651-1520 Fax: (916) 327-4478 | SB 503 |
THIRD READING
Bill No: SB 503
Author: Weber Pierson (D), et al.
Amended: 4/10/25
Vote: 21
SENATE HEALTH COMMITTEE: 10-0, 4/9/25
AYES: Menjivar, Durazo, Gonzalez, Grove, Limón, Padilla, Richardson, Rubio, Weber Pierson, Wiener
NO VOTE RECORDED: Valladares
SENATE JUDICIARY COMMITTEE: 13-0, 4/29/25
AYES: Umberg, Niello, Allen, Arreguín, Ashby, Caballero, Durazo, Laird, Stern, Valladares, Wahab, Weber Pierson, Wiener
SENATE APPROPRIATIONS COMMITTEE: 6-0, 5/23/25
AYES: Caballero, Seyarto, Cabaldon, Grayson, Richardson, Wahab
NO VOTE RECORDED: Dahle
SUBJECT: Health care services: artificial intelligence
SOURCE: Author
DIGEST:
This bill requires developers of patient care decision support tools, as defined, as well as health facilities, clinics, physician offices and other deployers of these tools, to make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool’s use in its health program or activities.
ANALYSIS:
Existing law:
- Establishes the Department of Health Care Access and Information (HCAI), and designates HCAI as the state agency designated to collect health facility data for use by all state agencies, including various financial data reports. [Health and Safety Code (HSC) §128730, et seq.]
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative AI to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative AI, and clear instructions on how to contact a human person. Defines “artificial intelligence,” for these purposes, as an engineered or machine-based system that varies in in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments. [HSC §1339.75]
- Requires the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, and state agency. Defines “high-risk automated decision system” as an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. [Government Code (GOV) §11546.45.5]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [Civil Code (CIV) §1798.100, et seq.]
- Establishes the Unruh Civil Rights Act that entitles all persons within the jurisdiction of this state to full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status. [CIV §51(b)]
This bill:
- Requires developers of patient care decision support tools, and health facilities, clinics, physician’s officers, or offices of a group practice, to have an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic.
- Requires developers and deployers, for each patient care decision support tool, to make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool’s use in its health program or activities.
- Requires developers to ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least once every three years.
- Defines various terms for purposes of this bill, including the following:
- “Biased impact,” means an unintended impact on an individual based on their protected characteristics;
- “Patient care decision support tool” means any automated or nonautomated tool, mechanism, method, technology, or combination thereof used by health facilities, clinics, physician’s offices, or offices of a group practice to support clinical decisionmaking in its health programs or activities.
- “Developer” means a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces a patient care decision support tool.
- “Deployer” means a person, partnership, state or local governmental agency, corporation, or developer that uses a patient care decision support tool.
- “Protected characteristic” means any characteristic listed in the Unruh Civil Rights Act, as specified.
- Specifies that a person, partnership, state or local governmental agency, or corporation can be both a developer and a deployer.
Comments
According to the author of this bill:
This bill is a crucial step toward ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. These technologies are becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill establishes a framework to identify and address biased outputs, ensuring that AI tools in healthcare are fair, accurate, and equitable for all patients. By taking a proactive approach, this bill helps protect patients from the effects of biased algorithms and ensures that technological advancements serve all communities equitably.
Background
What is AI? In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
According to an article in MIT News, from the Massachusetts Institute of Technology, before the generative AI boom of the past few years, when people talked about AI, typically they were talking about machine-learning models that can learn to make a prediction based on data. For instance, such models are trained, using millions of examples, to predict whether a certain X-ray shows signs of a tumor or if a particular borrower is likely to default on a loan. Generative AI can be thought of as a machine-learning model that is trained to create new data, rather than making a prediction about a specific dataset. A generative AI system is one that learns to generate more objects that look like the data it was trained on. Generative AI systems such as ChatGPT are powerful because they are trained on extremely large datasets, which could potentially take advantage of nearly all the information on the internet.
According to an article published by CalTech Science Exchange, today’s generative AI models produce content that often is indistinguishable from that created by humans. Generative AI has successfully written news articles, created realistic artwork, and composed aesthetically pleasing music. While these applications sometimes make glaring mistakes (sometimes referred to as hallucinations), they are being used for many purposes, including in health care.
Generative AI in health care. According to a July 25, 2023 viewpoint article in the Journal of the American Medical Association, “Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients,” generative AI is being heralded in the medical field for its potential to ease the long-lamented burden of medical documentation by generating visit notes, treatment codes, and medical summaries. This article stated that to date, no court in the U.S. has considered the question of liability for medical injuries caused by relying on AI-generated information. The article warned that to minimize risks, even medical professionals must exercise caution when relying on AI-generated information, due to the “black-box” nature of how these systems generate output, because it may be impossible for the physician to independently evaluate the accuracy of the AI’s output. A March 2024 article in Implement Science stated that the utility and impact of generative AI in health care remain poorly understood, with concerns about ethical and legal implications, integration into healthcare service delivery, and workforce utilization. The American Medical Association published “Principles for Augmented Intelligence Development, Deployment, and Use,” which was approved by the American Medical Association (AMA) Board of Trustees on November 14, 2023. According to this AMA document, as the number of AI-enabled health care tools and systems continue to grow, these technologies must be designed, developed, and deployed in a manner that is ethical, equitable, responsible, and transparent. The AMA notes that while the FDA regulates AI-enabled medical devices, many types of AI-enabled technologies fall outside the scope of FDA oversight, including AI that may have clinical applications, such as some clinical decision support functions. The AMA report specifically looked at transparency in use of AI-enabled systems, and stated that it is essential that use of AI in health care be transparent to both physicians and patients, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. The AMA report states that when AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients. The AMA noted that while transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. According to the AMA, when AI is used in a manner, which directly impacts patient care, access to care, or medical decision-making, that use of AI should be disclosed and documented to both physicians and patients. The opportunity for a patient to request additional review from a licensed clinician should be made available upon request. The AMA policy also states that AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
Racial bias in algorithms and AI. Several studies have identified risks of racial bias in algorithms and AI. A October 2019 article in the journal Science, “Dissecting racial bias in an algorithm used to manage the health of populations,” found that Black patients assigned the same level of risk by the algorithm are sicker than White patients. The study estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. According to the study, bias occurs because the algorithm uses health costs as a proxy for health needs, and because less money is spent on Black patients who have the same level of need, the algorithm falsely concludes that Black patients are healthier than equally sick White patients. The journal NPJ Digital Medicine published a study in 2023, “Large language models propagate race-based medicine,” that assessed four large language models used in healthcare systems. The study found that all models had examples of perpetuating race-based medicine in their response to questions. Models were not always consistent in their response when asked the same question repeatedly, and that based on their findings, these large language models could potentially cause harm by perpetuating debunked, racist ideas.
Trustworthy AI. The federal Health and Human Services Agency (HHS) has developed a “playbook” for AI development with the goal of maintaining public trust by using AI solutions that are “ethical, effective, and secure,” according to the HHS Chief AI Officer. With regard to HHS agencies, principles have been established for trustworthy AI that “fosters public trust and confidence while protecting privacy, civil rights, civil liberties, and American values, consistent with applicable laws.” The playbook describes AI as whether the solution or system:
- Performs tasks under varying and unpredictable circumstances without significant human oversight, or can learn from experience and improvement performance when exposed to data sets;
- Uses computer software, physical hardware, or other technology to solve tasks that require human-like perception, thinking, planning, learning, communication, or physical action;
- Thinks or acts like a human, including the use of cognitive architecture or neural networks (e.g., developed to mimic the underlying mechanisms of the human mind);
- Relies on a set of techniques, including machine learning, to approximate a cognitive task; and,
- Is designed to act rationally by utilizing intelligent software or an embodied robot to achieve goals using perception, planning, reasoning, learning, communicating, decision-making, and action.
The HHS principles for use of trustworthy AI in government cover six areas: fair and impartial application; transparent and explainable data use; responsible and accountable governance and policies; robust and reliable outputs; safe and secure from risks; and, that privacy should be respected, data not used beyond its intended and stated use, and that it’s use is approved by the data owner or steward.
Recent federal regulations on “patient care decision support tools.” One of the sections of the 2010 Affordable Care Act prohibited discrimination based on protected characteristics in certain health programs and activities, known as Section 1557. The Biden Administration, on May 6, 2024, published amendments to the implementing regulations of Section 1557 to extend this anti-discrimination provision to “patient care decision support tools.” These are defined as any automated or non-automated tools, mechanisms, methods, technologies, or combination thereof used by a covered entity (which includes providers that participate in Medi-Cal or Medicare) to support clinical decision-making in its health programs or activities. The regulations require a covered entity to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability. The regulations go on to require that for each patient care decision support tool that is identified, that the covered entity make reasonable efforts to mitigate the risk of discrimination resulting from the tool’s use in its health programs or activities.
Related/Prior legislation
AB 2930 (Bauer-Kahan of 2024) would have regulated the use of ADSs in order to prevent “algorithmic discrimination.” This includes requirements on developers and deployers that make and use these tools to make “consequential decisions” to perform impact assessments on ADSs. Would have also established the right of individuals to know when an ADS is being used, the right to opt out of its use, and an explanation of how it is used. AB 2930 died on the Assembly Floor.
AB 3030 (Calderon, Chapter 848, Statutes of 2024) requires a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence and clear instructions on how the patient may contact a human person.
SB 1120 (Becker, Chapter 879, Statutes of 2024) establishes requirements on health plans and insurers applicable to their use of AI for utilization review and utilization management decisions, including, that the use of AI, algorithm, or other software must be based upon a patient’s medical or other clinical history and individual clinical circumstances as presented by the requesting provider and not supplant health care provider decision making.
FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No
According to the Senate Appropriations Committee, unknown ongoing costs, likely hundreds of thousands, for CDPH for state administration (Licensing and Certification Fund).
SUPPORT: (Verified 5/8/25)
None received
OPPOSITION: (Verified 5/8/25)
None received
Prepared by: Vincent D. Marchand / HEALTH / (916) 651-4111
5/26/25 13:57:24
**** END ****
SENATE COMMITTEE ON APPROPRIATIONS
Senator Anna Caballero, Chair
2025 - 2026 Regular Session
SB 503 (Weber Pierson) - Health care services: artificial intelligence
Version: April 10, 2025 | Policy Vote: HEALTH 10 - 0, JUD. 13 - 0 |
Urgency: No | Mandate: No |
Hearing Date: May 12, 2025 | Consultant: Agnes Lee |
Bill Summary:
SB 503 would impose requirements on developers and deployers of patient care decision support tools, as specified.
Fiscal Impact:
Unknown ongoing costs, likely hundreds of thousands, for the California Department of Public Health (CDPH) for state administration (Licensing and Certification Fund).
Background:
The federal Affordable Care Act included provisions that prohibited discrimination based on protected characteristics in certain health programs and activities. On May 6, 2024, the federal government published regulations to extend this anti-discrimination provision to “patient care decision support tools.” These are defined as any automated or non-automated tools, mechanisms, methods, technologies, or combination thereof used by a covered entity (which includes providers that participate in Medi-Cal or Medicare) to support clinical decision-making in its health programs or activities. The regulations require a covered entity to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability. The regulations also require that for each patient care decision support tool that is identified, that the covered entity make reasonable efforts to mitigate the risk of discrimination resulting from the tool’s use in its health programs or activities.
Proposed Law:
Specific provisions of the bill would:
- Require developers of patient care decision support tools, and health facilities, clinics, physician’s offices, or offices of a group practice, to have an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic.
- Require, for each patient care decision support tool, developers/deployers to make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool’s use in its health programs or activities.
- Require developers to ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least once every three years.
- Specify that for purposes of these provisions, a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
-- END --
Senator Thomas Umberg, Chair
2025-2026 Regular Session
Bill No: SB 503 (Author:Weber Pierson)
Version: April 10, 2025
Hearing Date: April 29, 2025
Fiscal: Yes
Urgency: No
Consultant: CK
SUBJECT
Health care services: artificial intelligence
DIGEST
This bill imposes certain obligations with respect to preventing discrimination through use of patient care decision support tools, as defined, in healthcare settings.
EXECUTIVE SUMMARY
Section 1557 of the Affordable Care Act (ACA) prohibits discrimination on the basis of race, color, national origin, sex, age, or disability in certain health programs and activities receiving federal funds. The Secretary of Health and Human Services was authorized thereby to promulgate regulations to effectuate that edict. In May 2024, regulations were issued.
One of those regulations governed nondiscrimination in the use of patient care decision support tools. “Patient care decision support tool” means any automated or non-automated tool, mechanism, method, technology, or combination thereof used by a covered entity to support clinical decision-making in its health programs or activities. The regulation prohibits discrimination through the use of these tools. It further requires covered entities to make reasonable efforts to identify where the tools use inputs that measure certain characteristics, including race, color, national origin, sex, age, or disability, and to mitigate the risk of discrimination resulting from their use in health programs or activities.
This bill seeks to codify a version of this regulation in California law. It places a similar duty on developers of these tools, along with health facilities, clinics, physician’s offices, or offices of a group practice, to make reasonable efforts to identify input factors or variables that measure a protected category, and to mitigate the risk of discrimination on the basis of a protected characteristic by the tool. Developers are further required to test their tools for biased impacts every three years. The bill is author-sponsored. No timely support or opposition has been received. This bill passed out of the Senate Health Committee on a 10 to 0 vote.
PROPOSED CHANGES TO THE LAW
Existing federal law:
- Prohibits, pursuant to the ACA, an individual, on the ground prohibited under title VI of the Civil Rights Act of 1964, title IX of the Education Amendments of 1972, the Age Discrimination Act of 1975, or section 504 of the Rehabilitation Act of 1973, be excluded from participation in, be denied the benefits of, or be subjected to discrimination under, any health program or activity, any part of which is receiving Federal financial assistance, including credits, subsidies, or contracts of insurance, or under any program or activity that is administered by an Executive Agency or other entity, as provided. (42 U.S.C. § 18116 (“Section 1557”).)
- Provides that the enforcement mechanisms provided for and available under such title VI, title IX, section 504, and Age Discrimination Act shall apply for purposes of violations of the above. Authorizes the Secretary to promulgate regulations relevant thereto. (42 U.S.C. § 18116.)
- Provides the following regulatory guidelines with regards to the above:
- A covered entity must not discriminate on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools.
- A covered entity has an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability.
- For each patient care decision support tool identified in (b), a covered entity must make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities. (45 C.F.R. § 92.210.)
Existing state law:
- Establishes the Unruh Civil Rights Act (“Unruh”), which provides that all persons within the jurisdiction of this state are free and equal, and no matter what their sex, race, color, religion, ancestry, national origin, disability, medical condition, genetic information, marital status, sexual orientation, citizenship, primary language, or immigration status, are entitled to the full and equal accommodations, advantages, facilities, privileges, or services in all business establishments of every kind whatsoever. (Civ. Code § 51.)
- Requires the California Department of Technology (CDT) to conduct a comprehensive inventory of all high-risk automated decision systems (ADS) that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency. It defines the relevant terms:
- “Automated decision system” means a computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decisionmaking and materially impacts natural persons. “Automated decision system” does not include a spam email filter, firewall, antivirus software, identity and access management tools, calculator, database, dataset, or other compilation of data.
- “High-risk automated decision system” means an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. (Gov. Code § 11546.45.5.)
This bill:
- Defines “patient care decision support tool” to mean any automated or nonautomated tool, mechanism, method, technology, or combination thereof used by health facilities, clinics, physician’s offices, or offices of a group practice to support clinical decisionmaking in its health programs or activities.
- Provides that developers of patient care decision support tools, and health facilities, clinics, physician’s offices, or offices of a group practice, have an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic.
- Requires developers and deployers, for each patient care decision support tool, to make reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool’s use in its health programs or activities.
- Requires developers to ensure that patient care decision support tools are tested for biased impacts in the outputs produced by the tool at least once every three years.
- Provides a person, partnership, state or local governmental agency, or corporation may be both a developer and a deployer.
- Defines the relevant terms, including:
- “Biased impact” means an unintended impact on an individual based on their protected characteristics.
- “Protected characteristic” means a characteristic listed in subdivision (b) of Section 51 of the Civil Code.
COMMENTS
- ACA anti-discrimination regulations
Section 1557 is the civil rights provision of the ACA. Section 1557 prohibits discrimination on the grounds of race, color, national origin, sex, age, or disability in certain health programs and activities. Section 1557(c) of the ACA authorizes the Secretary of the Department of Health and Human Services to promulgate regulations to implement the nondiscrimination requirements of Section 1557.
Relevant here, such regulatory direction was issued last year with regard to “patient care decision support tools,” defined as any automated or non-automated tool, mechanism, method, technology, or combination thereof used by a covered entity to support clinical decision-making in its health programs or activities. (45 C.F.R. § 92.4.) The regulation provides:
- A covered entity must not discriminate on the basis of race, color, national origin, sex, age, or disability in its health programs or activities through the use of patient care decision support tools.
- A covered entity has an ongoing duty to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability.
- For each patient care decision support tool identified [above], a covered entity must make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities. (45 C.F.R. § 92.210.)
- Codifying federal regulations
This bill largely codifies this federal regulation. It imposes a duty on developers of patient care decision support tools, and health facilities, clinics, physician’s offices, or offices of a group practice, to make reasonable efforts to identify uses of patient care decision support tools in health programs or activities that employ input variables or factors that measure a protected characteristic. Developers and those deploying these tools are required to make “reasonable efforts to mitigate the risk of discrimination on the basis of a protected characteristic resulting from the tool’s use in its health programs or activities.”
In addition, and distinct from the federal regulation, the bill requires developers of these tools to ensure that the tools are tested for biased impacts in the outputs produced by the tool at least once every three years.
The bill defines “protected characteristic” as those laid out in Unruh and uses the definition of “patient care decision support tool” in the ACA regulations, but applying it to those tools used by health facilities, clinics, physician’s offices, or offices of a group practice, rather than “covered entities.”
The Federal Register discusses what is intended to be included in the term “patient care decision support tools,” including the following: ADS; AI; flowcharts; formulas; equations; calculators; algorithms; utilization management applications; software as medical devices (SaMDs); software in medical devices (SiMDs); screening, risk assessment, and eligibility tools; and diagnostic and treatment guidance tools. (89 F.R. § 37522.) Although such elaboration is not provided in this bill, the definition certainly encompasses a broad range of tools.
According to the author:
SB 503 is a crucial step towards ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. This technology is becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill will require collaboration between developers and healthcare facilities to identify AI tools used in the delivery of patient care and proactively work towards meaningfully reducing bias. By requiring routine testing and ongoing oversight, SB 503 will help promote safety, equity, and exceptional performance while protecting patients against avoidable harm.
SUPPORT
None received
OPPOSITION
None received
RELATED LEGISLATION
Pending Legislation:
SB 7 (McNerney, 2025) also regulates the use of ADS in the employment context, providing a minimum of 30 days’ notice to workers before deployment as well as post-deployment rights to notice, to correct information, and to appeal. It also restricts certain uses of ADS in the employment context. SB 7 is currently pending before this Committee and is set to be heard the same day as this bill.
SB 420 (Padilla, 2025) regulates the use of “high-risk ADS,” defined the same as high-risk AI systems in this bill. SB 420 includes requirements on developers and deployers to perform impact assessments on their systems. SB 420 establishes the right of individuals to know when an ADS is being used, details about the systems, and an opportunity to appeal ADS decisions, where technically feasible. SB 420 is currently in the Senate Appropriations Committee.
SB 468 (Becker, 2025) imposes a duty on a business that deploys a high-risk artificial intelligence system, or high-risk ADS, that processes personal information to protect that information and requires such a deployer to maintain a comprehensive information security program that meets specified requirements. SB 468 is currently in the Senate Appropriations Committee.
AB 1018 (Bauer-Kahan, 2025) regulates the development and deployment of ADS used to make consequential decisions, as defined. It requires a developer of a covered ADS to take certain actions, including conduct performance evaluations of the ADS, submit to third-party audits, and provide deployers to whom the developer transfers the covered ADS with certain information, including the results of those performance evaluations. AB 1018 is currently in the Assembly Judiciary Committee.
Prior Legislation:
SB 892 (Padilla, 2024) would have required CDT to develop and adopt regulations to create an ADS procurement standard, as specified, and prohibited a state agency from procuring ADS, entering into a contract for ADS, or any service that utilizes ADS, until CDT has adopted regulations creating an ADS procurement standard, as specified. SB 892 was vetoed by Governor Newsom, who stated in his veto message that aspects of the bill would disrupt ongoing work, “including existing information technology modernization efforts, which would lead to implementation delays and higher expenses for critical projects.”
AB 2885 (Bauer-Kahan, 2024) established a uniform definition for “artificial intelligence” in California’s code, which is used in this bill.
AB 302 (Ward, Ch. 800, Stats. 2023) required CDT, on or before September 1, 2024, to conduct a comprehensive inventory of all high-risk ADS that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, any state agency.
PRIOR VOTES:
Senate Health Committee (Ayes 10, Noes 0)
**************
SENATE COMMITTEE ON HEALTH
Senator Caroline Menjivar, Chair
BILL NO: SB 503
AUTHOR: | Weber Pierson | ||
VERSION: | March 25, 2025 | ||
HEARING DATE: | April 9, 2025 | ||
CONSULTANT: | Vincent D. Marchand | ||
SUBJECT: Health care services: artificial intelligence
SUMMARY:
Requires the Department of Health Care Access and Information and the Department of Technology to establish an advisory board related to the use of artificial intelligence (AI) in health care services to develop best practices for the use of AI models or systems by a health facility, clinic, physician’s office, or office of a group practice that uses AI in its provision of health care services and to develop a standardized testing system to test AI systems for biased impacts. Requires developers of AI systems to test for biased impacts in the outputs produced by AI systems.
Existing law:
- Establishes the Department of Health Care Access and Information (HCAI), and designates HCAI as the state agency designated to collect health facility data for use by all state agencies, including various financial data reports. [HSC §128730, et seq.]
- Requires a health facility, clinic, physician’s office, or office of a group practice that uses generative AI to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative AI, and clear instructions on how to contact a human person. Defines “artificial intelligence,” for these purposes, as an engineered or machine-based system that varies in in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments. [HSC §1339.75]
- Requires the Department of Technology to conduct, in coordination with other interagency bodies as it deems appropriate, a comprehensive inventory of all high-risk automated decision systems that have been proposed for use, development, or procurement by, or are being used, developed, or procured by, and state agency. Defines “high-risk automated decision system” as an automated decision system that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice. [GOV §11546.45.5]
- Establishes the California Privacy Protection Agency (CPPA) to implement and enforce the California Privacy Rights Act of 2020, and provides the CPPA with the full administrative power, authority and jurisdiction to implement and enforce the California Consumer Privacy Act of 2018, including responsibilities to update existing regulations and adopt new regulations. [CIV §1798.100, et seq.]
This bill:
- Requires HCAI and the Department of Technology to establish an advisory board related to the use of AI in health care services.
- Requires the advisory board to do all of the following:
- Develop best practices for the use of AI models or AI systems by a health facility, clinic, physician’s office, or office of a group practice that uses AI in its provision of health care services;
- Develop a standardized testing system with criteria for developers to test AI models or AI systems for biased impacts; and,
- Establish a statewide certificate that can be used to confirm that a developer’s version or release of their AI model or AI system meets standards set by the advisory board pursuant to the testing for biased impact.
- Defines “biased impact” as an unintended impact on an individual based on their protected characteristics.
- Requires developers of AI models or AI systems, in conjunction with health facilities, clinics, physician’s offices, or offices of a group practice, to test for biased impacts in the outputs produced by the specified AI model or AI system based on the health facility’s patient population.
- Defines “developer” as a person, partnership, state or local governmental agency, corporation, or deployer that designs, codes, substantially modifies, or otherwise produces an AI model or AI system that generates an output that can influence physical or virtual environments. Defines “deployer” as a person, partnership, state or local governmental agency, corporation, or developer that uses a covered AI model or AI system that generates an output that can influence physical or virtual environments. Specifies that a person, partnership, state or local governmental agency, or corporation can be a developer and a deployer if the person or entity both designs, codes, substantially modifies, or otherwise produces an AI model or AI system, and also uses a covered AI model or AI system.
- Requires developers to use an existing testing system designated by the advisory board until the advisory board has developed its standardized testing system pursuant to 2b) above, and permits developers to alternatively use the board’s testing system after the advisory board has developed its testing system.
- Permits developers, after the advisory board has created the certification in 2c) above, to use the advisory board’s standardize testing system to certify their AI models or AI systems.
FISCAL EFFECT:
This bill has not been analyzed by a fiscal committee.
COMMENTS:
- Author’s statement. According to the author, this bill is a crucial step toward ensuring fairness in healthcare by addressing the racial biases embedded in AI models and systems. These technologies are becoming more prevalent in healthcare, yet research has shown that these systems can produce biased outputs that disproportionately affect communities of color. Without proper oversight, these biases can go unchecked, deepening existing disparities in our healthcare system. This bill takes a stand against the harmful impacts of such biases by establishing an expert advisory board tasked with developing a standardized, comprehensive testing system. This framework will identify and address biased outputs, ensuring that AI tools in healthcare are fair, accurate, and equitable for all patients. By taking a proactive approach, this bill helps protect patients from the effects of biased algorithms and ensures that technological advancements serve all communities equitably.
- What is AI? In a draft regulation, the CPPA defines AI as follows: AI means a machine-based system that infers, from the input it receives, how to generate outputs that can influence physical or virtual environments. The AI may do this to achieve explicit or implicit objectives. Outputs can include predictions, content, recommendations, or decisions. Different AI varies in its levels of autonomy and adaptiveness after deployment. For example, AI includes generative AI models, such as large language models (LLMs), that can learn from inputs and create new outputs, such as text, images, audio, or video; and facial- or speech-recognition or -detection technology.
According to an article in MIT News, from the Massachusetts Institute of Technology, before the generative AI boom of the past few years, when people talked about AI, typically they were talking about machine-learning models that can learn to make a prediction based on data. For instance, such models are trained, using millions of examples, to predict whether a certain X-ray shows signs of a tumor or if a particular borrower is likely to default on a loan. Generative AI can be thought of as a machine-learning model that is trained to create new data, rather than making a prediction about a specific dataset. A generative AI system is one that learns to generate more objects that look like the data it was trained on. Generative AI systems such as ChatGPT are powerful because they are trained on extremely large datasets, which could potentially take advantage of nearly all the information on the internet.
According to an article published by CalTech Science Exchange, today’s generative AI models produce content that often is indistinguishable from that created by humans. Generative AI has successfully written news articles, created realistic artwork, and composed aesthetically pleasing music. While these applications sometimes make glaring mistakes (sometimes referred to as hallucinations), they are being used for many purposes, including in health care.
- Generative AI in health care. According to a July 25, 2023 viewpoint article in the Journal of the American Medical Association, “Generative AI in Health Care and Liability Risks for Physicians and Safety Concerns for Patients,” generative AI is being heralded in the medical field for its potential to ease the long-lamented burden of medical documentation by generating visit notes, treatment codes, and medical summaries. This article stated that to date, no court in the U.S. has considered the question of liability for medical injuries caused by relying on AI-generated information. The article warned that to minimize risks, even medical professionals must exercise caution when relying on AI-generated information, due to the “black-box” nature of how these systems generate output, because it may be impossible for the physician to independently evaluate the accuracy of the AI’s output. A March 2024 article in Implement Science stated that the utility and impact of generative AI in health care remain poorly understood, with concerns about ethical and legal implications, integration into healthcare service delivery, and workforce utilization. The American Medical Association published “Principles for Augmented Intelligence Development, Deployment, and Use,” which was approved by the AMA Board of Trustees on November 14, 2023. According to this AMA document, as the number of AI-enabled health care tools and systems continue to grow, these technologies must be designed, developed, and deployed in a manner that is ethical, equitable, responsible, and transparent. The AMA notes that while the FDA regulates AI-enabled medical devices, many types of AI-enabled technologies fall outside the scope of FDA oversight, including AI that may have clinical applications, such as some clinical decision support functions. The AMA report specifically looked at transparency in use of AI-enabled systems, and stated that it is essential that use of AI in health care be transparent to both physicians and patients, and disclosure should contribute to physician and patient knowledge and not create unnecessary administrative burden. The AMA report states that when AI is utilized in health care decision-making, that use should be disclosed and documented in order to limit risks to, and mitigate inequities for, both physicians and patients. The AMA noted that while transparency does not necessarily ensure AI-enabled tools are accurate, secure, or fair, it is difficult to establish trust if certain characteristics are hidden. According to the AMA, when AI is used in a manner, which directly impacts patient care, access to care, or medical decision-making, that use of AI should be disclosed and documented to both physicians and patients. The opportunity for a patient to request additional review from a licensed clinician should be made available upon request. The AMA policy also states that AI tools or systems cannot augment, create, or otherwise generate records, communications, or other content on behalf of a physician without that physician’s consent and final review.
- Racial bias in algorithms and AI. Several studies have identified risks of racial bias in algorithms and AI. A October 2019 article in the journal Science, “Dissecting racial bias in an algorithm used to manage the health of populations,” found that Black patients assigned the same level of risk by the algorithm are sicker than White patients. The study estimated that this racial bias reduces the number of Black patients identified for extra care by more than half. According to the study, bias occurs because the algorithm uses health costs as a proxy for health needs, and because less money is spent on Black patients who have the same level of need, the algorithm falsely concludes that Black patients are healthier than equally sick White patients. The journal NPJ Digital Medicine published a study in 2023, “Large language models propagate race-based medicine,” that assessed four large language models used in healthcare systems. The study found that all models had examples of perpetuating race-based medicine in their response to questions. Models were not always consistent in their response when asked the same question repeatedly, and that based on their findings, these large language models could potentially cause harm by perpetuating debunked, racist ideas.
- Trustworthy AI. The federal Health and Human Services Agency (HHS) has developed a “playbook” for AI development with the goal of maintaining public trust by using AI solutions that are “ethical, effective, and secure,” according to the HHS Chief AI Officer. With regard to HHS agencies, principles have been established for trustworthy AI that “fosters public trust and confidence while protecting privacy, civil rights, civil liberties, and American values, consistent with applicable laws.” The playbook describes AI as whether the solution or system:
- Performs tasks under varying and unpredictable circumstances without significant human oversight, or can learn from experience and improvement performance when exposed to data sets;
- Uses computer software, physical hardware, or other technology to solve tasks that require human-like perception, thinking, planning, learning, communication, or physical action;
- Thinks or acts like a human, including the use of cognitive architecture or neural networks (e.g., developed to mimic the underlying mechanisms of the human mind);
- Relies on a set of techniques, including machine learning, to approximate a cognitive task; and,
- Is designed to act rationally by utilizing intelligent software or an embodied robot to achieve goals using perception, planning, reasoning, learning, communicating, decision-making, and action.
The HHS principles for use of trustworthy AI in government cover six areas: fair and impartial application; transparent and explainable data use; responsible and accountable governance and policies; robust and reliable outputs; safe and secure from risks; and, that privacy should be respected, data not used beyond its intended and stated use, and that it’s use is approved by the data owner or steward.
- Recent federal regulations on “patient care decision support tools.” One of the sections of the 2010 Affordable Care Act prohibited discrimination based on protected characteristics in certain health programs and activities, known as Section 1557. The Biden Administration, on May 6, 2024, published amendments to the implementing regulations of Section 1557 to extend this anti-discrimination provision to “patient care decision support tools.” These are defined as any automated or non-automated tools, mechanisms, methods, technologies, or combination thereof used by a covered entity (which includes providers that participate in Medi-Cal or Medicare) to support clinical decision-making in its health programs or activities. The regulations require a covered entity to make reasonable efforts to identify uses of patient care decision support tools in its health programs or activities that employ input variables or factors that measure race, color, national origin, sex, age, or disability. The regulations go on to require that for each patient care decision support tool that is identified, that the covered entity make reasonable efforts to mitigate the risk of discrimination resulting from the tool’s use in its health programs or activities.
- Double referral. This bill is double referred. Should it pass out of this Committee, it will be referred to the Senate Committee on Governmental Organization.
- Related legislation. SB 7 (McNerney) requires an employer, or a vendor engaged by the employer, to provide a written notice that an automated decision system (ADS), for the purpose of making employment-related decisions, is in use at the workplace to a worker who will be directly or indirectly affected by the ADS, as specified. Defines ADS as any computational process derived from machine learning, statistical modeling, data analytics, or artificial intelligence that issues simplified output, including a score, classification, or recommendation, that is used to assist or replace human discretionary decision-making and materially impacts natural persons. SB 7 is set for hearing on April 9, 2025 in the Senate Labor, Public Employment and Retirement Committee.
SB 243 (Padilla and Becker) establishes specified requirements on persons who make a companion chatbot that uses an artificial intelligence system with a natural language interface, as specified, including taking reasonable steps to prevent a companion chatbot from encouraging increased engagement, usage, or response rates. Among other provisions, SB 243 requires an operator of a companion chatbot platform to annually report to the Department of Health Care Services on the number of times the operator has detected exhibitions of suicidal ideation by users, and the number of times a companion chatbot brought up suicidal ideation or actions with the users. SB 243 is set for hearing on April 8, 2025 in the Senate Judiciary Committee.
SB 420 (Padilla) establishes requirements for high-risk ADSs, including requiring developers of systems made available for use after January 1, 2026, to perform an impact assessment on the system before making the system publicly available for use. Defines a high-risk ADS as an ADS that is used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that material impact access to, or approval for, any of the following: education enrollment or opportunity; employment or employment opportunity; essential utilities; temporary, short-term, or long-term housing; health care services; lending services; a legal right or service; or, an essential government service. SB 420 is for hearing on April 8, 2025 in the Senate Judiciary Committee.
SB 468 (Becker) requires a business that deploys high-risk artificial intelligence systems that process personal information to develop, implement, and maintain a comprehensive information security program that is written in one or more readily accessible parts and contains administrative, technical, and physical safeguards, as specified. SB 468 is set for hearing on April 22, 2025 in the Senate Judiciary Committee.
SB 813 (McNerney) establishes a process for the Attorney General to designate “multistakeholder regulatory organizations” to certify that developers of artificial intelligence models or applications exercise heightened care and compliance with best practices for the prevention of personal injury and property damage, among other obligations. SB 813 is pending in the Senate Judiciary Committee.
AB 1018 (Bauer-Kahan) regulates the use of ADSs used to make consequential decisions, as defined, and would require a developer of a covered ADS to take certain actions, including conducting performance evaluations that assess whether the ADS is intended to treat individuals or groups of individuals diffrently on the basis of a protected characteristic. AB 1018 is set for hearing on April 22, 2025 in the Assembly Privacy and Consumer Protection Committee.
- Prior legislation. AB 2930 (Bauer-Kahan of 2024) would have regulated the use of ADSs in order to prevent “algorithmic discrimination.” This includes requirements on developers and deployers that make and use these tools to make “consequential decisions” to perform impact assessments on ADSs. Would have also established the right of individuals to know when an ADS is being used, the right to opt out of its use, and an explanation of how it is used. AB 2930 died on the Assembly Floor.
AB 3030 (Calderon, Chapter 848, Statutes of 2024) requires a health facility, clinic, physician’s office, or office of a group practice that uses generative artificial intelligence to generate written or verbal patient communications pertaining to patient clinical information to ensure that those communications include a disclaimer that indicates to the patient that the communication was generated by generative artificial intelligence and clear instructions on how the patient may contact a human person.
SB 1120 (Becker, Chapter 879, Statutes of 2024) establishes requirements on health plans and insurers applicable to their use of AI for utilization review and utilization management decisions, including, that the use of AI, algorithm, or other software must be based upon a patient’s medical or other clinical history and individual clinical circumstances as presented by the requesting provider and not supplant health care provider decision making. - Policy comments.
- Definition of AI is broad. The requirements in this bill apply to developers that design, code, substantially modify, or otherwise produce “an AI model or AI system that generates an output that can influence physical or virtual environments.” While this definition is similar to some existing definitions, including a recently enacted provision of law requiring health care providers to disclose when a patient communication was generated by AI, this definition has the potential to have a broader sweep than intended by the author. The author may wish to consider utilizing the definition of “patient care decision support tools,” used in Section 1557 federal regulations, which are “any automated or non-automated tools, mechanisms, methods, technologies, or combination thereof used…to support clinical decision-making” in health programs.
- Testing requirements are unclear. This bill requires the advisory board established by this bill to develop a standardized testing system for developers to test AI systems for biased impacts. Developers are required to “use an existing testing system designated by the advisory board until the advisory board has developed its standardized testing system.” Therefore, until the advisory board is established, reviews testing systems, and then designates testing systems, it is unclear how a developer will be able to meet the testing requirements in this bill.
SUPPORT AND OPPOSITION:
Support: None received
Oppose: None received
-- END --
Discussed in Hearing
Senate Floor
Assembly Floor
Assembly Standing Committee on Privacy and Consumer Protection
Assembly Standing Committee on Health
Senate Standing Committee on Judiciary
Senate Standing Committee on Health
Bill Author
Bill Co-Author(s):