Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, as defined.

This bill, the California Digital Content Provenance Standards, would require a generative artificial intelligence (AI) provider, as provided, to, among other things, apply provenance data to synthetic content produced or significantly modified by a generative AI system that the provider makes available, as those terms are defined, and to conduct adversarial testing exercises, as prescribed. The bill would prohibit, among other things, providers and distributors of software and online services from making available a system, application, tool, or service that is designed for the primary purpose of removing provenance data from synthetic content, as provided.

This bill would require a newly manufactured recording device sold, offered for sale, or distributed in California to offer users the option to apply difficult to remove provenance data to nonsynthetic content produced by that device and would require the application of that provenance data to be compatible with state-of-the-art adopted and relevant industry standards. If technically feasible and secure, the bill would require a recording device manufacturer to offer a software or firmware update enabling a user of a recording device manufactured before July 1, 2026, and purchased in California to apply difficult to remove provenance data to the nonsynthetic content created by the device and decode any provenance data attached to nonsynthetic content created by the device.

This bill would require a large online platform, as defined, capable of disseminating specified content to use labels to disclose, as specified, any machine-readable provenance data detected in synthetic content that is distributed on its platform. If content uploaded to or distributed on a large online platform by a user does not contain specified provenance data or if the contents provenance data cannot be interpreted or detected, the bill would require a large online platform to label the content as having unknown provenance. The bill would require a large online platform to use a visual disclosure that contains specified information, including the copyrightholder or licensor information, when labeling and disclosing provenance data of sound recordings and music videos.

Beginning July 1, 2026, and annually thereafter, this bill would require a large online platform to produce a transparency report that identifies moderation of deceptive synthetic content on their platform and would authorize that report to include, among other things, instances where synthetic or potentially deceptive content was identified and removed by the platform, as applicable.

This bill would authorize the Department of Technology (department) to assess specified administrative penalties for prescribed violations of the bills provisions, including an administrative penalty of up to $500,000 $100,000 for each violation that is intentional or is the result of grossly negligent conduct, to be deposited in the Digital Content Provenance Administrative Fund which the bill would establish in the State Treasury. The bill would, upon appropriation by the Legislature for this express purpose, authorize the expenditure of moneys in the fund by the department to administer these provisions.

This bill would make its provisions operative on July 1, 2026.

Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, as defined.

This bill would authorize the department Department of Technology (department) to assess specified administrative penalties for prescribed violations of the bills provisions, including an administrative penalty of up to $500,000 for each violation that is intentional or is the result of grossly negligent conduct. The bill would require the department to adopt regulations as necessary to implement and carry out the purposes of this act and to review and update those regulations as needed. conduct, to be deposited in the Digital Content Provenance Administrative Fund which the bill would establish in the State Treasury. The bill would, upon appropriation by the Legislature for this express purpose, authorize the expenditure of moneys in the fund by the department to administer these provisions.

This bill would make its provisions operative on July 1, 2026.

Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, as defined.

This bill, the California Provenance, Authenticity and Watermarking Standards Act, would require a generative artificial intelligence (AI) provider to, among other things, place an imperceptible and maximally indelible watermark into synthetic content produced or significantly modified by a generative AI system that the provider makes available, as those terms are defined. The bill would require, within 96 hours of discovering a material vulnerability or failure in a generative AI system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, a generative AI provider to report the vulnerability or failure to the Department of Technology and to notify other generative AI providers, as specified. The bill would also require a conversational AI system, as defined, to clearly and prominently disclose to users that the conversational AI system generates synthetic content.

This bill would require a large online platform, as defined, to, among other things, use labels to disclose the provenance data of content distributed on its platform, as specified. If a large online platform is not able to detect the provenance data of content, the bill would require the platform to label the content as unknown provenance. If content uploaded to or distributed on a large online platform by a user does not contain provenance data, or if the contents provenance data cannot be interpreted or detected by the platform, the bill would require the platform to require the user to disclose whether the content is synthetic content, as specified.

This bill would require newly manufactured recording devices sold, offered for sale, or distributed in California to offer users the option to place a watermark into content produced by that device. The bill would require the watermark to be compatible with widely used industry standards. If technically feasible, the bill would require a recording device manufacturer, as defined, to offer to a user of a recording device purchased in California before January July 1, 2026, a software or firmware update enabling the user to place a watermark on the content created by the device and decode the provenance data.

Beginning January 1, 2026, July 1, 2026, and annually thereafter, this bill would also require generative AI providers and large online platforms to produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated in their generative AI systems or hosted on their generative AI hosting platforms, as prescribed. The bill would require the report to be audited by qualified, independent auditors who are required to assess and either validate or invalidate the claims made in the report, as specified.

This bill would provide that a violation of its provisions may result in an administrative penalty, assessed by the department, of up to $1,000,000 or 5% of the violators annual global revenue, whichever is greater. authorize the department to assess specified administrative penalties for prescribed violations of the bills provisions, including an administrative penalty of up to $500,000 for each violation that is intentional or is the result of grossly negligent conduct. The bill would require the department to adopt regulations as necessary to implement and carry out the purposes of this act and to review and update those regulations as needed.

Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, as defined.

Beginning February 1, 2025, this bill, the California Provenance, Authenticity and Watermarking Standards Act, would require a generative artificial intelligence (AI) provider to, among other things, place an imperceptible and maximally indelible watermarks containing provenance data watermark into synthetic content produced or significantly modified by a generative AI system that the provider makes available, as those terms are defined. The bill would require, within 24 96 hours of discovering a material vulnerability or failure in a generative AI system, system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, a generative AI provider to report the vulnerability or failure to the Department of Technology and to notify other generative AI providers, as specified. The bill would also require a conversational AI system, as defined, to clearly and prominently disclose to users that the conversational AI system receives generates synthetic content.

Beginning March 1, 2025, this bill would require a large online platform, as defined, to, among other things, use labels to prominently disclose the provenance data found in watermarks or digital signatures in content distributed to users on its platforms, of content distributed on its platform, as specified. The bill would require a large online platform to require a user that uploads or distributes content on its platform to disclose whether the content is synthetic content, as specified. If a large online platform is not able to detect the provenance data of content, the bill would require the platform to label the content as unknown provenance. If content uploaded to or distributed on a large online platform by a user does not contain provenance data, or if the contents provenance data cannot be interpreted or detected by the platform, the bill would require the platform to require the user to disclose whether the content is synthetic content, as specified.

Beginning January 1, 2026, this bill would require newly manufactured digital cameras and recording devices sold, offered for sale, or distributed in California to offer users the option to place an authenticity watermark and provenance watermark in the a watermark into content produced by that device. The bill would require the authenticity watermark and provenance watermark to be compatible with widely used industry standards. If technically feasible, the bill would require a camera and recording device manufacturer, as defined, to offer to a user of a digital camera or recording device purchased in California prior to before January 1, 2026, a software or firmware update enabling the user to place an authenticity watermark and provenance a watermark on the content created by the device. device and decode the provenance data.

Beginning January 1, 2026, and annually thereafter, this bill would also require generative AI providers and large online platforms to produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated in their generative AI systems or hosted on their generative AI hosting platforms, as prescribed. The bill would require the report to be audited by qualified, independent auditors who are required to assess and either validate or invalidate the claims made in the report, as specified.

This bill would provide that a violation of its provisions may result in an administrative penalty, assessed by the department, of up to $1,000,000 or 5% of the violators annual global revenue, whichever is greater. The bill would require the department to adopt regulations as necessary to implement and carry out the purposes of this act and to review and update those regulations as needed.

Existing law requires the Secretary of Government Operations to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance. For the purpose of informing that coordinated plan, existing law requires the secretary to evaluate, among other things, the impact of the proliferation of deepfakes, as defined.

Beginning February 1, 2025, this bill, the California Provenance, Authenticity and Watermarking Standards Act, would require a generative artificial intelligence (AI) system provider, as defined, to take certain actions to assist in the disclosure of provenance data to mitigate harms caused by inauthentic content, including placing imperceptible and maximally indelible watermarks containing provenance data into content created by an AI system that the generative AI system provider makes available. provider to, among other things, place imperceptible and maximally indelible watermarks containing provenance data into synthetic content produced or significantly modified by a generative AI system that the provider makes available, as those terms are defined. The bill would require, within 24 hours of discovering a vulnerability or failure in an a generative AI system, a generative AI system provider to report the vulnerability or failure to the Department of Technology and to notify other generative AI system providers, as specified. The bill would also require a conversational AI system, as defined, to clearly and prominently disclose to users that the conversational AI system receives synthetic content.

Beginning March 1, 2025, this bill would require a large online platform, as defined, to, among other things, use labels to prominently disclose the provenance data found in watermarks or digital signatures in content distributed to users on its platforms, as specified. The bill would require a large online platform to use state-of-the-art techniques, including, but not limited to, analysis of user behavioral signals indicating usage of synthetic content, to detect and label inauthentic text content that is uploaded or distributed by individual users or networks of users. require a user that uploads or distributes content on its platform to disclose whether the content is synthetic content, as specified.

Beginning January 1, 2026, this bill would require newly manufactured digital cameras and recording devices sold, offered for sale, or distributed in California to offer users the option to place an authenticity watermark and provenance watermark in the content produced by that device. The bill would require the authenticity watermark and provenance watermark to be compatible with widely used industry standards, as specified. If a digital camera or recording device purchased in California prior to January 1, 2026, is capable of receiving a software or firmware update that would enable a user to place an authenticity watermark and provenance watermark on the content created by the device, the bill would require the devices manufacturer to offer that update. standards. If technically feasible, the bill would require a camera and recording device manufacturer, as defined, to offer to a user of a digital camera or recording device purchased in California prior to January 1, 2026, a software or firmware update enabling the user to place an authenticity watermark and provenance watermark on the content created by the device.

Beginning January 1, 2026, and annually thereafter, this bill would also require specified entities generative AI providers and large online platforms to produce a Risk Assessment and Mitigation Report that assesses the risks posed by synthetic content and the harms that have been or could be caused by synthetic content, and harms caused by synthetic content generated in their systems or hosted on their platforms, as prescribed. The bill would require the report to be audited by qualified, independent auditors who are required to assess and either validate or invalidate the claims made in the report, as specified.

This bill would provide that a violation of its provisions may result in an administrative penalty, assessed by the department, of up to $1,000,000 or 5% of the violators annual global revenue, whichever is greater. The bill would require the department to adopt regulations as necessary to implement and carry out the purposes of this act and to review and update those regulations as needed.

Existing law requires an operator of a commercial internet website or online service that collects personally identifiable information through the internet about individual consumers residing in California who use or visit its commercial internet website or online service to conspicuously post its privacy policy on its internet website, as prescribed.

This bill would make nonsubstantive changes to that provision.

 THIRD READING  

Bill No: AB 3211

Author: Wicks (D)

Amended: 8/23/24 in Senate

Vote: 21

SENATE JUDICIARY COMMITTEE: 8-2, 6/18/24

AYES: Umberg, Allen, Ashby, Caballero, Durazo, Roth, Stern, Wahab

NOES: Wilk, Niello

NO VOTE RECORDED: Laird

SENATE GOVERNMENTAL ORG. COMMITTEE: 10-2, 6/25/24

AYES: Dodd, Alvarado-Gil, Archuleta, Ashby, Bradford, Glazer, Padilla, Roth, Rubio, Smallwood-Cuevas

NOES: Wilk, Seyarto

NO VOTE RECORDED: Jones, Nguyen, Ochoa Bogh, Portantino

SENATE APPROPRIATIONS COMMITTEE: 4-2, 8/15/24

AYES: Caballero, Ashby, Becker, Wahab

NOES: Jones, Seyarto

NO VOTE RECORDED: Bradford

ASSEMBLY FLOOR: 62-0, 5/22/24 - See last page for vote

SUBJECT: California Digital Content Provenance Standards

SOURCE: California Initiative for Technology & Democracy

DIGEST:

This bill establishes the California Digital Content Provenance Standards Act, which requires a generative AI (GenAI) provider to, among other things, take certain actions to assist in the disclosure of provenance data. This bill requires an online platform, as defined, to, among other things, use labels to disclose provenance data found in synthetic content, as specified. This bill requires platforms to produce transparency reports. The bill requires recording device manufacturers to enable options for embedding provenance data into recordings.

Senate Floor Amendments of 8/22/24 substantially narrowed and restructured the bill to loosen the requirements being applied to developers and online platforms with respect to detecting and labeling the digital provenance of content, as provided.

Senate Floor Amendments of 8/23/24 provide an exception for providers to refrain from sharing reports with academic institutions, lower the administrative penalties, and rework several provisions.

ANALYSIS:

Existing law:

1. Defines “deepfake” as audio or visual content that has been generated or manipulated by artificial intelligence which would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do without their consent. (Gov't Code § 11547.5.)
2. Defines “digital content forgery” as the use of technologies, including artificial intelligence and machine learning techniques, to fabricate or manipulate audio, visual, or text content with the intent to mislead. (Gov't Code § 11547.5.)
3. Defines “digital content provenance” as the verifiable chronology of the original piece of digital content, such as an image, video, audio recording, or electronic document. (Gov't Code § 11547.5.)

This bill:

1. Defines the relevant terms.
2. Requires a GenAI provider whose GenAI system is capable of producing digital content that would falsely appear to a reasonable person to depict real-life persons, objects, places, entities, or events to apply provenance data, as specified, that is difficult to remove or disassociate. The GenAI provider is further required to make available to the public a provenance detection tool, as provided.
3. Requires a GenAI provider to conduct adversarial testing exercises following relevant guidelines established by the National Institute on Standards and Technology (NIST) to assess the robustness of provenance data methods and whether the systems can be used to add false provenance data, as specified.
4. Requires newly manufactured recording devices sold, offered for sale, or distributed in California to offer users the option to apply difficult to remove provenance data to nonsynthetic content produced by that device.
5. Requires the recording devices to clearly inform users of the existence of the settings upon a user’s first use of the recording function and to contain a clear indicator that provenance data is being applied.
6. Requires, if technically feasible, a recording device manufacturer to offer a software or firmware update enabling a user of a recording device manufactured before July 1, 2026, and purchased in California, to apply provenance data to the content created by the device and to decode the provenance data.
7. Requires a large online platform to use labels to disclose the provenance data of synthetic content distributed on its platform, as specified.
8. Provides that if content uploaded to or distributed on a large online platform by a user does not contain provenance data or if the content’s provenance data cannot be interpreted or detected by the platform using technically feasible methods, a large online platform shall label the content as having unknown provenance.
9. Requires large online platforms to produce a transparency report that identifies moderation of deceptive synthetic content on their platform.
10. Authorizes CDT to assess an administrative penalty, as provided.
11. Provides an operative date of July 1, 2026.
12. Includes a severability clause.

Background

Certain forms of media – audio recordings, video recordings, and still images – can be powerful evidence of the truth. While such media have always been susceptible to some degree of manipulation, fakes were relatively easy to detect. The rapid advancement of AI technology, specifically the wide-scale introduction of GenAI models, has made it drastically cheaper and easier to produce synthetic content, audio, images, text, and video recordings that are not real, but that are so realistic that they are virtually impossible to distinguish from authentic content, including so-called “deepfakes.”

Among other things, this bill works to ensure that providers of these GenAI systems are applying provenance data to synthetic content produced or significantly modified by their systems and equipping consumers with a tool to identify when specific content has been generated by their systems; places requirements on large online platforms to disclose the provenance data of content and to detect and label specified content; and requires recording device manufacturers to allow for the embedding of provenance data on nonsynthetic content produced by their devices. This bill is sponsored by the California Initiative for Technology & Democracy and supported by a wide variety of groups, including SEIU California and NextGen CA. This bill is opposed by Oakland Privacy and a coalition of industry groups, including Netchoice.

[For a more thorough analysis of the bill, please see the Senate Judiciary Committee analysis.]

Comment

According to the author:

The primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. Specifically, the bill aims to address the interrelated problems stemming from the increasing proliferation and sophistication of generative AI technologies that can create synthetic or "deepfake" content that is difficult to distinguish from human-generated, non-synthetic content . . . .

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Senate Appropriations Committee:

Unknown, potentially significant costs (General Fund) to the California Department of Technology (CDT) in the tens of millions of dollars annually to implement the requirements of this bill. CDT indicates that this work is outside of the scope of CDT’s current technical team and would require additional administrative, technical, and research expertise. To build this new business capability, CDT would need 14 new General Fund positions, two temporary program managers to establish the new team, and 12 ongoing staff to manage and resource this program.

CDT notes the importance of integrating artificial intelligence talent, including GenAI talent, in-house to ensure proper oversight in responsible use of GenAI across strategy, design, and technical functions within all our services and offerings. Foundational estimates include primarily full-time employees, with an additional request for funding to source and award contracts for external consultants necessary to provide GenAI expertise in the short-term within defined scopes of work to be determined by CDT. In parallel, a recruitment strategy specifically built to attract and retain GenAI talent is necessary to meet increased demand for GenAI expertise within and outside of the CDT. Accordingly, CDT would require the following resources:

• Year 1: 2 positions and $500,000, $600,000 External Consultants, and $500,000 GenAI Talent Practices; and,
• Year 2 and ongoing: 12 positions and $2,100,000, $600,000 External Consultants, and $400,000 GenAI Talent Practices.

CDT also notes that, as GenAI is still in its infancy in integrating with public sector services, the State will need to explore different methods of attracting talent while competing with private sector compensations. Private sector salaries for GenAI talent range from two to three times the level of the State's current maximum salary range. In addition to salaries, private sector total compensation can include bonuses, equity, and stock options that can further widen the gap between public and private sector total compensation packages, making it even more difficult for the State to be financially competitive for the same talent pool.

The fiscal estimate currently accounts for hiring talent with some fundamental skills in the GenAI space while leveraging GenAI Talent Practice funds to upskill and train the workforce in areas relevant to public service. If the State does not successfully recruit sufficient resources or provide adequate training to confidentially implement requirements within this bill as it is currently written, costs to hire similar headcounts and skillsets of external consultants are expected to be two to three times the cost of hiring talents at the State's current salary.

SUPPORT: (Verified 8/25/24)

California Initiative for Technology & Democracy (source)

Accountable Tech

Adobe

Bay Rising

California Voter Foundation

Catalyst California

Center for Countering Digital Hate

Check My Ads

Chinese Progressive Association

City and County of San Francisco, Board of Supervisors

Digimarc Corporation

Hmong Innovating Politics

Move (Mobilize, Organize, Vote, Empower) the Valley

NextGen California

OpenAI

Partnership for the Advancement of New Americans

SEIU California

Steg.AI

Techequity Action

OPPOSITION: (Verified 8/25/24)

California Chamber of Commerce

Computer and Communications Industry Association

Netchoice

Oakland Privacy

Technet

ARGUMENTS IN SUPPORT: The California Initiative for Technology & Democracy writes:

The detrimental impact of these AI tools represents a monumental challenge for our society that will require multi-faceted solutions that recognize and respond to the dynamic nature of evolving technology. The EU’s new AI Act leads the way, by requiring companies to develop content authentication and provenance mechanisms to tackle AI-generated disinformation.6 Domestic efforts have been weaker. Federal action has been limited to the Biden Administration’s executive order on AI, which tasks the Department of Commerce to develop guidance for content authentication.7 Industry efforts and voluntary commitments have also cropped up to tackle this problem.8

California has the opportunity to lead the U.S. response to these threats. As the home of Silicon Valley and the heartland of AI innovation, California can and must play an important role in shaping the solution.

To that end, AB 3211 represents a phased-in solution to address this complex problem that complements existing standard-setting efforts. The bill has been developed with insights from EU regulators who have worked intimately on the AI Act and from federal policy experts who see, in the face of congressional inaction, California’s power to drive nationwide change.

ARGUMENTS IN OPPOSITION: A coalition in opposition, including TechNet and NetChoice, write:

In its standards for large online platforms, AB 3211 should more clearly delineate between 1st party and 3rd party content. 1st party content would be images, videos, or audio that is generated using a large online platform’s generative AI tools and is then posted or distributed on that platform. In this instance, a platform can actually control the creation of a content provenance or watermark into the content. As mentioned, many of our companies are already working to incorporate this type of technology to increase transparency around AI-generated content. It is currently technically infeasible to accurately and reliably detect content that is created using a different platform’s AI tools. As noted above, considering the current ease with which current watermarks can be broken, a legal requirement and mandate for 3rd party content isn’t appropriate.

ASSEMBLY FLOOR: 62-0, 5/22/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Davies, Dixon, Mike Fong, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Ortega, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Rodriguez, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Valencia, Villapudua, Waldron, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas

NO VOTE RECORDED: Calderon, Cervantes, Megan Dahle, Essayli, Flora, Vince Fong, Friedman, Gallagher, Holden, Hoover, Lackey, Mathis, Jim Patterson, Joe Patterson, Luz Rivas, Sanchez, Ta, Wallis

Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113

8/27/24 20:51:54

**** END ****

 THIRD READING  

Bill No: AB 3211

Author: Wicks (D)

Amended: 8/22/24 in Senate

Vote: 21

SENATE JUDICIARY COMMITTEE: 8-2, 6/18/24

AYES: Umberg, Allen, Ashby, Caballero, Durazo, Roth, Stern, Wahab

NOES: Wilk, Niello

NO VOTE RECORDED: Laird

SENATE GOVERNMENTAL ORG. COMMITTEE: 10-2, 6/25/24

AYES: Dodd, Alvarado-Gil, Archuleta, Ashby, Bradford, Glazer, Padilla, Roth, Rubio, Smallwood-Cuevas

NOES: Wilk, Seyarto

NO VOTE RECORDED: Jones, Nguyen, Ochoa Bogh, Portantino

SENATE APPROPRIATIONS COMMITTEE: 4-2, 8/15/24

AYES: Caballero, Ashby, Becker, Wahab

NOES: Jones, Seyarto

NO VOTE RECORDED: Bradford

ASSEMBLY FLOOR: 62-0, 5/22/24 - See last page for vote

SUBJECT: California Digital Content Provenance Standards

SOURCE: California Initiative for Technology & Democracy

DIGEST:

This bill establishes the California Digital Content Provenance Standards Act, which requires a generative AI (GenAI) provider to, among other things, take certain actions to assist in the disclosure of provenance data. This bill requires an online platform, as defined, to, among other things, use labels to disclose provenance data found in synthetic content, as specified. This bill requires platforms to produce transparency reports. The bill requires recording device manufacturers to enable options for embedding provenance data into recordings.

Senate Floor Amendments of 8/22/24 substantially narrow and restructure the bill to loosen the requirements being applied to developers and online platforms with respect to detecting and labeling the digital provenance of content, as provided.

ANALYSIS:

Existing law:

1. Defines “deepfake” as audio or visual content that has been generated or manipulated by artificial intelligence which would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do without their consent. (Gov't Code § 11547.5.)
2. Defines “digital content forgery” as the use of technologies, including artificial intelligence and machine learning techniques, to fabricate or manipulate audio, visual, or text content with the intent to mislead. (Gov't Code § 11547.5.)
3. Defines “digital content provenance” as the verifiable chronology of the original piece of digital content, such as an image, video, audio recording, or electronic document. (Gov't Code § 11547.5.)

This bill:

1. Defines the relevant terms.
2. Requires a GenAI provider whose GenAI system is capable of producing digital content that would falsely appear to a reasonable person to depict real-life persons, objects, places, entities, or events to apply provenance data, as specified, that is difficult to remove. The GenAI provider is further required to make available to the public a provenance detection tool, as provided.
3. Requires a GenAI provider to conduct adversarial testing exercises following relevant guidelines established by the National Institute on Standards and Technology (NIST) to assess the robustness of provenance data methods and whether the systems can be used to add false provenance data, as specified.
4. Requires newly manufactured recording devices sold, offered for sale, or distributed in California to offer users the option to apply difficult to remove provenance data to nonsynthetic content produced by that device.
5. Requires the recording devices to clearly inform users of the existence of the settings upon a user’s first use of the recording function and to contain a clear indicator that provenance data is being applied.
6. Requires, if technically feasible, a recording device manufacturer to offer a software or firmware update enabling a user of a recording device manufactured before July 1, 2026, and purchased in California, to apply provenance data to the content created by the device and to decode the provenance data.
7. Requires a large online platform to use labels to disclose the provenance data of synthetic content distributed on its platform, as specified.
8. Provides that if content uploaded to or distributed on a large online platform by a user does not contain provenance data or if the content’s provenance data cannot be interpreted or detected by the platform using technically feasible methods, a large online platform shall label the content as having unknown provenance.
9. Requires large online platforms to produce a transparency report that identifies moderation of deceptive synthetic content on their platform.
10. Authorizes CDT to assess an administrative penalty, as provided.
11. Provides an operative date of July 1, 2026.
12. Includes a severability clause.

Background

Certain forms of media – audio recordings, video recordings, and still images – can be powerful evidence of the truth. While such media have always been susceptible to some degree of manipulation, fakes were relatively easy to detect. The rapid advancement of AI technology, specifically the wide-scale introduction of GenAI models, has made it drastically cheaper and easier to produce synthetic content, audio, images, text, and video recordings that are not real, but that are so realistic that they are virtually impossible to distinguish from authentic content, including so-called “deepfakes.”

Among other things, this bill works to ensure that providers of these GenAI systems are applying provenance data to synthetic content produced or significantly modified by their systems and equipping consumers with a tool to identify when specific content has been generated by their systems, places requirements on large online platforms to disclose the provenance data of content and to detect and label specified content, and requires recording device manufacturers to allow for the embedding of provenance data on nonsynthetic content produced by their devices. This bill is sponsored by the California Initiative for Technology & Democracy and supported by a wide variety of groups, including SEIU California and NextGen CA. This bill is opposed by Oakland Privacy and a coalition of industry groups, including Netchoice.

[For a more thorough analysis of the bill, please see the Senate Judiciary Committee analysis.]

Comment

According to the author:

The primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. Specifically, the bill aims to address the interrelated problems stemming from the increasing proliferation and sophistication of generative AI technologies that can create synthetic or "deepfake" content that is difficult to distinguish from human-generated, non-synthetic content . . . .

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Senate Appropriations Committee:

Unknown, potentially significant costs (General Fund) to the California Department of Technology (CDT) in the tens of millions of dollars annually to implement the requirements of this bill. CDT indicates that this work is outside of the scope of CDT’s current technical team and would require additional administrative, technical, and research expertise. To build this new business capability, CDT would need 14 new General Fund positions, two temporary program managers to establish the new team, and 12 ongoing staff to manage and resource this program.

CDT notes the importance of integrating artificial intelligence talent, including GenAI talent, in-house to ensure proper oversight in responsible use of GenAI across strategy, design, and technical functions within all our services and offerings. Foundational estimates include primarily full-time employees, with an additional request for funding to source and award contracts for external consultants necessary to provide GenAI expertise in the short-term within defined scopes of work to be determined by CDT. In parallel, a recruitment strategy specifically built to attract and retain GenAI talent is necessary to meet increased demand for GenAI expertise within and outside of the CDT. Accordingly, CDT would require the following resources:

• Year 1: 2 positions and $500,000, $600,000 External Consultants, and $500,000 GenAI Talent Practices; and,
• Year 2 and ongoing: 12 positions and $2,100,000, $600,000 External Consultants, and $400,000 GenAI Talent Practices.

CDT also notes that, as GenAI is still in its infancy in integrating with public sector services, the State will need to explore different methods of attracting talent while competing with private sector compensations. Private sector salaries for GenAI talent range from two to three times the level of the State's current maximum salary range. In addition to salaries, private sector total compensation can include bonuses, equity, and stock options that can further widen the gap between public and private sector total compensation packages, making it even more difficult for the State to be financially competitive for the same talent pool.

The fiscal estimate currently accounts for hiring talent with some fundamental skills in the GenAI space while leveraging GenAI Talent Practice funds to upskill and train the workforce in areas relevant to public service. If the State does not successfully recruit sufficient resources or provide adequate training to confidentially implement requirements within this bill as it is currently written, costs to hire similar headcounts and skillsets of external consultants are expected to be two to three times the cost of hiring talents at the State's current salary.

SUPPORT: (Verified 8/15/24)

California Initiative for Technology & Democracy (source)

Accountable Tech

Bay Rising

California Voter Foundation

Catalyst California

Center for Countering Digital Hate

Check My Ads

Chinese Progressive Association

City and County of San Francisco, Board of Supervisors

Hmong Innovating Politics

Move (Mobilize, Organize, Vote, Empower) the Valley

NextGen California

Partnership for the Advancement of New Americans

SEIU California

Steg.AI

Techequity Action

OPPOSITION: (Verified 8/15/24)

California Chamber of Commerce

Computer and Communications Industry Association

Netchoice

Oakland Privacy

Technet

ARGUMENTS IN SUPPORT: The California Initiative for Technology & Democracy writes:

The detrimental impact of these AI tools represents a monumental challenge for our society that will require multi-faceted solutions that recognize and respond to the dynamic nature of evolving technology. The EU’s new AI Act leads the way, by requiring companies to develop content authentication and provenance mechanisms to tackle AI-generated disinformation.6 Domestic efforts have been weaker. Federal action has been limited to the Biden Administration’s executive order on AI, which tasks the Department of Commerce to develop guidance for content authentication.7 Industry efforts and voluntary commitments have also cropped up to tackle this problem.8

California has the opportunity to lead the U.S. response to these threats. As the home of Silicon Valley and the heartland of AI innovation, California can and must play an important role in shaping the solution.

To that end, AB 3211 represents a phased-in solution to address this complex problem that complements existing standard-setting efforts. The bill has been developed with insights from EU regulators who have worked intimately on the AI Act and from federal policy experts who see, in the face of congressional inaction, California’s power to drive nationwide change.

ARGUMENTS IN OPPOSITION: A coalition in opposition, including TechNet and NetChoice, write:

In its standards for large online platforms, AB 3211 should more clearly delineate between 1st party and 3rd party content. 1st party content would be images, videos, or audio that is generated using a large online platform’s generative AI tools and is then posted or distributed on that platform. In this instance, a platform can actually control the creation of a content provenance or watermark into the content. As mentioned, many of our companies are already working to incorporate this type of technology to increase transparency around AI-generated content. It is currently technically infeasible to accurately and reliably detect content that is created using a different platform’s AI tools. As noted above, considering the current ease with which current watermarks can be broken, a legal requirement and mandate for 3rd party content isn’t appropriate.

ASSEMBLY FLOOR: 62-0, 5/22/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Davies, Dixon, Mike Fong, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Ortega, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Rodriguez, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Valencia, Villapudua, Waldron, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas

NO VOTE RECORDED: Calderon, Cervantes, Megan Dahle, Essayli, Flora, Vince Fong, Friedman, Gallagher, Holden, Hoover, Lackey, Mathis, Jim Patterson, Joe Patterson, Luz Rivas, Sanchez, Ta, Wallis

Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113

8/23/24 21:36:37

**** END ****

 THIRD READING  

Bill No: AB 3211

Author: Wicks (D)

Amended: 6/24/24 in Senate

Vote: 21

SENATE JUDICIARY COMMITTEE: 8-2, 6/18/24

AYES: Umberg, Allen, Ashby, Caballero, Durazo, Roth, Stern, Wahab

NOES: Wilk, Niello

NO VOTE RECORDED: Laird

SENATE GOVERNMENTAL ORG. COMMITTEE: 10-2, 6/25/24

AYES: Dodd, Alvarado-Gil, Archuleta, Ashby, Bradford, Glazer, Padilla, Roth, Rubio, Smallwood-Cuevas

NOES: Wilk, Seyarto

NO VOTE RECORDED: Jones, Nguyen, Ochoa Bogh, Portantino

SENATE APPROPRIATIONS COMMITTEE: 4-2, 8/15/24

AYES: Caballero, Ashby, Becker, Wahab

NOES: Jones, Seyarto

NO VOTE RECORDED: Bradford

ASSEMBLY FLOOR: 62-0, 5/22/24 - See last page for vote

SUBJECT: California Provenance, Authenticity and Watermarking Standards

SOURCE: California Initiative for Technology & Democracy

DIGEST:

This bill establishes the California Provenance, Authenticity and Watermarking Standards Act, which requires a generative AI (GenAI) system provider to, among other things, take certain actions to assist in the disclosure of provenance data. This bill requires an online platform, as defined, to, among other things, use labels to prominently disclose the provenance data found in watermarks or digital signatures in content, as specified, and to use state-of-the-art techniques to detect and label inauthentic text content that is uploaded or distributed by users. This bill requires GenAI providers and platforms to produce annual impact assessment reports. The bill requires recording device manufacturers to enable options for embedding provenance data into recordings.

ANALYSIS:

Existing law:

1. Defines “deepfake” as audio or visual content that has been generated or manipulated by artificial intelligence which would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do without their consent. (Gov't Code § 11547.5.)
2. Defines “digital content forgery” as the use of technologies, including artificial intelligence and machine learning techniques, to fabricate or manipulate audio, visual, or text content with the intent to mislead. (Gov't Code § 11547.5.)
3. Defines “digital content provenance” as the verifiable chronology of the original piece of digital content, such as an image, video, audio recording, or electronic document. (Gov't Code § 11547.5.)

This bill:

1. Defines the relevant terms.
2. Requires a GenAI provider to place an imperceptible and maximally indelible watermark into synthetic content produced or significantly modified by the provider’s GenAI system, as provided. To the greatest extent possible, watermarks shall be designed to communicate information that identifies content as synthetic and identifies the provider in the event that a sample of synthetic content is corrupted, downscaled, cropped, or otherwise damaged.
3. Requires a GenAI provider to make available to the public a watermark decoder that is easy to use and adheres to relevant national or international standards.
4. Requires a GenAI provider to conduct AI red-teaming exercises involving third-party experts to test whether watermarks can be easily removed or whether the provider’s GenAI systems can be used to falsely add watermarks to otherwise nonsynthetic content. Red-teaming exercises shall be conducted before the release of any new Gen AI system and annually thereafter, as provided. A provider shall make summaries of these exercises publicly available, as specified, and full reports are to be sent to CDT. Details that pose an immediate risk to public safety or provide information that could be used to disable or circumvent the functionality of watermarks must be removed.
5. Requires a GenAI system capable of producing potentially deceptive content to generate and store, in a searchable online database in a manner that can be retrieved by a viewer of the content, a digital fingerprint of and provenance data for any piece of potentially deceptive content that they produce. This provenance shall not include personally identifiable information.
6. Prohibits providers and distributors of software and online services from making available a system, application, tool, or service that is designed to remove watermarks from synthetic content.
7. Prohibits a GenAI hosting platform from making available a GenAI system that does not place maximally indelible watermarks, as required herein.
8. Requires a GenAI provider to, within 96 hours of discovering a material vulnerability or failure in a GenAI system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, report the vulnerability or failure to CDT, as provided. Providers must also notify other providers that may be affected by similar vulnerabilities or failures.
9. Imposes specified notice requirements on providers.
10. Requires a conversational AI system to clearly and prominently disclose to users that the conversational AI system generates synthetic content, as specified. In all conversational interfaces of such a system, the system shall, at the beginning of a user’s interaction, obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system before beginning the conversation, as specified.
11. Requires newly manufactured recording devices sold, offered for sale, or distributed in California to offer users the option to place a watermark into content produced by that device. A user shall have the option to customize the types of provenance data communicated by these watermarks, including by removing any personally identifiable information (PII). PII cannot be included by default.
12. Requires the recording devices to clearly inform users of the existence of the watermark settings upon a user’s first use of the recording function and to contain a clear indicator that a watermark is being applied. A watermark shall, if enabled, be applied to nonsynthetic content produced using third-party applications that bypass default recording applications in order to offer recording functionalities and must be compatible with widely used industry standards.
13. Requires, if technically feasible, a recording device manufacturer to offer a software or firmware update enabling a user of a recording device manufactured before January 1, 2026, and purchased in California to place a watermark on the content created by the device and to decode the provenance data.
14. Requires a large online platform to use labels to disclose the provenance data of content distributed on its platform, as specified. The labels shall prominently display whether content is fully synthetic, partially synthetic, nonsynthetic, nonsynthetic with minor modifications, or does not contain a watermark. A user shall be able to click or tap on a label to inspect provenance data in an easy-to-understand format.
15. Requires a large online platform to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by GenAI systems without watermarking functionality. If the platform is not able to detect the provenance data of content, then the platform shall label the content as unknown provenance.
16. Provides that if content uploaded to or distributed on a platform does not contain provenance data, or if the content’s provenance data cannot be interpreted or detected by the platform, a platform shall require the user to disclose whether the content is synthetic content or if they are uncertain. A large online platform shall include prominent warnings to users that uploading or distributing synthetic content without disclosing that it is synthetic content may be a violation of platform policy.
17. Requires a large online platform to use state-of-the-art techniques to detect and label text-based potentially deceptive content that is uploaded by users.
18. Requires a large online platform to make accessible some functionality for users to apply a digital signature to nonsynthetic content, and include options that do not require disclosure of PII.
19. Requires a large online platform that can detect potentially deceptive content that does not contain watermarks that comply with applicable industry standards to generate and store, in an online database to be shared and made privately accessible by all other online platforms and CDT, digital fingerprints and any associated provenance data for these images. This provenance data shall not include PII.
20. Authorizes CDT to share access to these databases with coordinating bodies acting to facilitate more rapid and computationally efficient detection and labeling of synthetic content.
21. Requires GenAI providers and large online platforms to produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated by their GenAI systems or hosted on their GenAI hosting platforms, as provided.
22. Authorizes CDT to assess an administrative penalty, as provided.
23. Requires CDT, within 90 days of the effective date of this bill, to adopt regulations to implement and carry out the purposes of this bill. CDT is required to review and update its regulations as needed, including adopting specific national or international standards for provenance, authenticity, watermarking, and digital signatures, as long as the standards do not weaken these provisions.
24. Provides an operative date of July 1, 2026.
25. Includes a severability clause.

Background

Certain forms of media – audio recordings, video recordings, and still images – can be powerful evidence of the truth. While such media have always been susceptible to some degree of manipulation, fakes were relatively easy to detect. The rapid advancement of AI technology, specifically the wide-scale introduction of GenAI models, has made it drastically cheaper and easier to produce synthetic content, audio, images, text, and video recordings that are not real, but that are so realistic that they are virtually impossible to distinguish from authentic content, including so-called “deepfakes.”

Among other things, this bill works to ensure that providers of these GenAI systems are equipping consumers with a tool to identify when specific content has been generated by their systems, places requirements on large online platforms to disclose the provenance data of content and to detect and label inauthentic content, and requires recording device manufacturers to allow for the embedding of watermarks on recordings. This bill is sponsored by the California Initiative for Technology & Democracy and supported by a wide variety of groups, including SEIU California and NextGen CA. The bill is opposed by Oakland Privacy and a coalition of industry groups, including Netchoice. For a more thorough analysis of the bill, please see the Senate Judiciary Committee analysis.

Comment

According to the author:

The primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. Specifically, the bill aims to address the interrelated problems stemming from the increasing proliferation and sophistication of generative AI technologies that can create synthetic or "deepfake" content that is difficult to distinguish from human-generated, non-synthetic content . . . .

FISCAL EFFECT: Appropriation: No Fiscal Com.: Yes Local: No

According to the Senate Appropriations Committee:

Unknown, potentially significant costs (General Fund) to the California Department of Technology (CDT) in the tens of millions of dollars annually to implement the requirements of this bill. CDT indicates that this work is outside of the scope of CDT’s current technical team and would require additional administrative, technical, and research expertise. To build this new business capability, CDT would need 14 new General Fund positions, two temporary program managers to establish the new team, and 12 ongoing staff to manage and resource this program.

CDT notes the importance of integrating artificial intelligence talent, including GenAI talent, in-house to ensure proper oversight in responsible use of GenAI across strategy, design, and technical functions within all our services and offerings. Foundational estimates include primarily full-time employees, with an additional request for funding to source and award contracts for external consultants necessary to provide GenAI expertise in the short-term within defined scopes of work to be determined by CDT. In parallel, a recruitment strategy specifically built to attract and retain GenAI talent is necessary to meet increased demand for GenAI expertise within and outside of the CDT. Accordingly, CDT would require the following resources:

• Year 1: 2 positions and $500,000, $600,000 External Consultants, and $500,000 GenAI Talent Practices; and,
• Year 2 and ongoing: 12 positions and $2,100,000, $600,000 External Consultants, and $400,000 GenAI Talent Practices.

CDT also notes that, as GenAI is still in its infancy in integrating with public sector services, the State will need to explore different methods of attracting talent while competing with private sector compensations. Private sector salaries for GenAI talent range from two to three times the level of the State's current maximum salary range. In addition to salaries, private sector total compensation can include bonuses, equity, and stock options that can further widen the gap between public and private sector total compensation packages, making it even more difficult for the State to be financially competitive for the same talent pool.

The fiscal estimate currently accounts for hiring talent with some fundamental skills in the GenAI space while leveraging GenAI Talent Practice funds to upskill and train the workforce in areas relevant to public service. If the State does not successfully recruit sufficient resources or provide adequate training to confidentially implement requirements within this bill as it is currently written, costs to hire similar headcounts and skillsets of external consultants are expected to be two to three times the cost of hiring talents at the State's current salary.

SUPPORT: (Verified 8/15/24)

California Initiative for Technology & Democracy (source)

Accountable Tech

Bay Rising

California Voter Foundation

Catalyst California

Center for Countering Digital Hate

Check My Ads

Chinese Progressive Association

City and County of San Francisco, Board of Supervisors

Hmong Innovating Politics

Move (Mobilize, Organize, Vote, Empower) the Valley

NextGen California

Partnership for the Advancement of New Americans

SEIU California

Steg.AI

Techequity Action

OPPOSITION: (Verified 8/15/24)

California Chamber of Commerce

Computer and Communications Industry Association

Netchoice

Oakland Privacy

Technet

ARGUMENTS IN SUPPORT: The California Initiative for Technology & Democracy writes:

The detrimental impact of these AI tools represents a monumental challenge for our society that will require multi-faceted solutions that recognize and respond to the dynamic nature of evolving technology. The EU’s new AI Act leads the way, by requiring companies to develop content authentication and provenance mechanisms to tackle AI-generated disinformation.6 Domestic efforts have been weaker. Federal action has been limited to the Biden Administration’s executive order on AI, which tasks the Department of Commerce to develop guidance for content authentication.7 Industry efforts and voluntary commitments have also cropped up to tackle this problem.8

California has the opportunity to lead the U.S. response to these threats. As the home of Silicon Valley and the heartland of AI innovation, California can and must play an important role in shaping the solution.

To that end, AB 3211 represents a phased-in solution to address this complex problem that complements existing standard-setting efforts. The bill has been developed with insights from EU regulators who have worked intimately on the AI Act and from federal policy experts who see, in the face of congressional inaction, California’s power to drive nationwide change.

ARGUMENTS IN OPPOSITION: A coalition in opposition, including TechNet and NetChoice, write:

In its standards for large online platforms, AB 3211 should more clearly delineate between 1st party and 3rd party content. 1st party content would be images, videos, or audio that is generated using a large online platform’s generative AI tools and is then posted or distributed on that platform. In this instance, a platform can actually control the creation of a content provenance or watermark into the content. As mentioned, many of our companies are already working to incorporate this type of technology to increase transparency around AI-generated content. It is currently technically infeasible to accurately and reliably detect content that is created using a different platform’s AI tools. As noted above, considering the current ease with which current watermarks can be broken, a legal requirement and mandate for 3rd party content isn’t appropriate.

ASSEMBLY FLOOR: 62-0, 5/22/24

AYES: Addis, Aguiar-Curry, Alanis, Alvarez, Arambula, Bains, Bauer-Kahan, Bennett, Berman, Boerner, Bonta, Bryan, Juan Carrillo, Wendy Carrillo, Chen, Connolly, Davies, Dixon, Mike Fong, Gabriel, Garcia, Gipson, Grayson, Haney, Hart, Irwin, Jackson, Jones-Sawyer, Kalra, Lee, Low, Lowenthal, Maienschein, McCarty, McKinnor, Muratsuchi, Stephanie Nguyen, Ortega, Pacheco, Papan, Pellerin, Petrie-Norris, Quirk-Silva, Ramos, Rendon, Reyes, Rodriguez, Blanca Rubio, Santiago, Schiavo, Soria, Ting, Valencia, Villapudua, Waldron, Ward, Weber, Wicks, Wilson, Wood, Zbur, Robert Rivas

NO VOTE RECORDED: Calderon, Cervantes, Megan Dahle, Essayli, Flora, Vince Fong, Friedman, Gallagher, Holden, Hoover, Lackey, Mathis, Jim Patterson, Joe Patterson, Luz Rivas, Sanchez, Ta, Wallis

Prepared by: Christian Kurpiewski / JUD. / (916) 651-4113

8/18/24 13:37:25

**** END ****

SENATE COMMITTEE ON APPROPRIATIONS

Senator Anna Caballero, Chair

2023 - 2024 Regular Session

AB 3211 (Wicks) - California Provenance, Authenticity and Watermarking Standards

Bill Summary:

AB 3211 requires generative artificial intelligence (GenAI) providers to watermark synthetic content; requires large online platforms to disclose provenance data or label content of unknown origins, as specified; and, imposes penalties for non-compliance.

Fiscal Impact:

Unknown, potentially significant costs (General Fund) to the California Department of Technology (CDT) in the tens of millions of dollars annually to implement the requirements of this bill. CDT indicates that this work is outside of the scope of CDT’s current technical team and would require additional administrative, technical, and research expertise. To build this new business capability, CDT would need 14 new General Fund positions, two temporary program managers to establish the new team, and 12 ongoing staff to manage and resource this program.

CDT notes the importance of integrating artificial intelligence talent, including GenAI talent, in-house to ensure proper oversight in responsible use of GenAI across strategy, design, and technical functions within all our services and offerings. Foundational estimates include primarily full-time employees, with an additional request for funding to source and award contracts for external consultants necessary to provide GenAI expertise in the short-term within defined scopes of work to be determined by CDT. In parallel, a recruitment strategy specifically built to attract and retain GenAI talent is necessary to meet increased demand for GenAI expertise within and outside of the CDT. Accordingly, CDT would require the following resources:

• Year 1: 2 positions and $500,000, $600,000 External Consultants, and $500,000 GenAI Talent Practices; and,
• Year 2 and ongoing: 12 positions and $2,100,000, $600,000 External Consultants, and $400,000 GenAI Talent Practices.

CDT also notes that, as GenAI is still in its infancy in integrating with public sector services, the State will need to explore different methods of attracting talent while competing with private sector compensations. Private sector salaries for GenAI talent range from two to three times the level of the State's current maximum salary range. In addition to salaries, private sector total compensation can include bonuses, equity, and stock options that can further widen the gap between public and private sector total compensation packages, making it even more difficult for the State to be financially competitive for the same talent pool.

The fiscal estimate currently accounts for hiring talent with some fundamental skills in the GenAI space while leveraging GenAI Talent Practice funds to upskill and train the workforce in areas relevant to public service. If the State does not successfully recruit sufficient resources or provide adequate training to confidentially implement requirements within this bill as it is currently written, costs to hire similar headcounts and skillsets of external consultants are expected to be two to three times the cost of hiring talents at the State's current salary.

Background:

Provenance, in relation to GenAI content, refers to the detailed history and origin of the synthetic content generated by an AI system. This includes information about the creation process, modifications, and the AI system responsible for generating or altering the content. Provenance can ensure transparency and traceability, helping users verify the authenticity of the content and its source. Although images have long been manipulated or faked, the proliferation of AI systems that can generate and modify images and videos has dramatically increased the amount of convincing but ultimately fake content circulating in the public domain. This emerging technology has already resulted in creation of nonconsensual pornography, scams targeting individuals and corporations, and political propaganda and misinformation.

Proposed Law:

• Requires, by July 1, 2026, a GenAI provider to do all of the following:
  • Place an imperceptible and maximally indelible watermark into synthetic content produced or significantly modified by a GenAI system that the provider makes available, as specified.
  • Make available to the public a watermark decoder that is easy to use by an individual seeking to quickly assess the provenance of a single piece of content, and adheres, to the greatest extent possible, to relevant national or international standards.
  • Conduct AI red-teaming exercises, as defined, involving third-party experts to test whether watermarks can be easily removed from synthetic content produced by the provider’s GenAI systems, as specified. Red-teaming exercises shall be conducted before the release of any new GenAI system and annually thereafter.
• Requires a GenAI system capable of producing potentially deceptive content, as defined, to generate and store, in a searchable online database, a digital fingerprint of and provenance data for any piece of potentially deceptive content that they produce, as specified.
• Prohibits providers and distributors of software and online services from making available a system, application, tool, or service that is designed to remove watermarks from synthetic content.
• Prohibits GenAI hosting platforms from making available a GenAI system that does not place maximally indelible watermarks communicating provenance data into content created or substantially modified by the system in a manner consistent with specifications pursuant to above.
• Requires a GenAI provider, within 96 hours of discovering a material vulnerability or failure in a GenAI system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, to report the vulnerability to the California Department of Technology (CDT).
• Requires a provider to notify other GenAI providers that may be affected by similar vulnerabilities, or failures in a manner that allows the other GenAI provider to harden their own AI systems against similar risks, as specified.
• Requires a provider to use commercially reasonable efforts to notify affected parties, including, but not limited to, online platforms, researchers or users who received incorrect results from a watermark decoder, or users who produced AI content that contained incorrect or insufficient provenance data, as specified.
• Requires a provider to make any report to the CDT publicly available in a location linked from the home page of the provider’s internet website with a clearly labeled link that has a similar look, feel, and size relative to other links on the same web page, as specified.
• Requires a conversational AI system to clearly and prominently disclose to users that the conversational AI system generates synthetic content, as specified.
• Requires, in all conversational interfaces of a conversational AI system, the conversational AI system to, at the beginning of the user’s interaction with the system, to obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system, as specified.
• Requires a newly manufactured recording device sold, offered for sale, or distributed in California to offer users the option to place a watermark into content produced by that device, as specified.
• Requires a recording device manufacturer, if technically feasible, to offer a software or firmware update enabling a user of a recording device manufactured before July 1, 2026, and purchased in California, to do both of the following:
  • Place a watermark on the content created by the device.
  • Decode the provenance data.
• Requires a large online platform to use labels to disclose the provenance data of content distributed on its platform, as specified.
• Requires the disclosure to be readily legible to an average viewer or, if the content is in audio format, to be clearly audible, as specified.
• Requires a large online platform to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by GenAI systems without watermarking functionality, as specified.
• Provides that if content uploaded to or distributed on a large online platform by a user does not contain provenance data, or if the content’s provenance data cannot be interpreted or detected by the platform, the platform must require the user to disclose whether the content is synthetic content, as specified.
• Requires a large online platform to use state-of-the-art techniques to detect and label text-based potentially deceptive content that is uploaded by users.
• Requires a large online platform to make accessible some functionality for users to apply a digital signature to non-synthetic content. The functionality shall include options that do not require disclosure of personally identifiable information.
• Requires a large online platform that can detect potentially deceptive content that does not contain watermarks that comply with applicable industry standards to generate and store, in an online database to be shared and privately accessible by all other online platforms and the CDT, digital fingerprints and any associated provenance data for these images, as specified.
• Requires GenAI providers, beginning July 1, 2026, and annually thereafter, to produce a Risk Assessment and Mitigation Report that assess the risks posed and harms caused by synthetic content generated by their GenAI systems or hosted on their GenAI hosting platforms.
• Requires the report to include, but not be limited to, assessments of the distribution of illegal GenAI-generated child sexual abuse materials (CSAM), nonconsensual intimate imagery, disinformation related to elections or public health, plagiarism, or other instances where synthetic or potentially deceptive content caused or may have the potential to cause harm.
• Requires the report to incorporate information known to the GenAI provider or large online platform about known harms caused by synthetic content generated by their systems or hosted on their platforms, as informed by reports submitted to, and confirmed by, the provider or platform, and independent investigation as appropriate, including, for example, illegal material.
• Requires the report to be audited by qualified, independent auditors who shall assess and either validate or invalidate the claims made in the report, as specified.
• Authorizes the CDT to assess an administrative penalty as follows:
  • If a violation of this bill is intentional or is the result of grossly negligent conduct, a penalty of up to $500,000 for each violation.
  • If a violation of this bill is not intentional or is not the result of grossly negligent conduct, a penalty of up to $50,000 for each violation.
• Requires CDT, within 90 days of the date upon which this bill takes effect, to adopt regulations to implement and carry out the purposes of this bill, as specified.
• Includes related Legislative findings and declarations.

-- END --

SENATE COMMITTEE ON GOVERNMENTAL ORGANIZATION

Senator Bill Dodd

Chair

2023 - 2024 Regular

Bill No: Hearing Date: 6/25/2024

SUBJECT:

DIGEST: This bill, the California Provenance, Authenticity, and Watermarking Standards (PAWS) Act, requires generative artificial intelligence (GenAI) providers, among other things, to watermark synthetic content, as specified; requires large online platforms, among other things, to disclose provenance data or label content of unknown origins, as specified; and imposes penalties for non-compliance, as specified. This bill would become operative on July 1, 2026.

ANALYSIS:

Existing law:

1. Requires the Secretary of the Government Operations Agency (GovOps) to develop a coordinated plan to, among other things, investigate the feasibility of, and obstacles to, developing standards and technologies for state departments to determine digital content provenance.
2. Requires, for the purpose of informing that coordinated plan, the Secretary of GovOps to evaluate, among other things, the impact of the proliferation of deep fakes, as specified.
3. Defines “social media platform” to mean a public or semipublic internet-based service or application that has users in California and a substantial function of the service or application is to connect users in order to allow users to interact socially with each other within the service or application and the service or application allows users to construct a public or semipublic profile, as specified.
4. The Administrative Procedures Act (APA) governs the procedure for the adoption, amendment, or repeal of regulations by state agencies and for the review of those regulatory actions by the Office of Administrative Law (OAL).

This bill:

1. Requires, by July 1, 2026, a GenAI provider to do all of the following:
   1. Place an imperceptible and maximally indelible watermark into synthetic content produced or significantly modified by a GenAI system that the provider makes available, as specified.
   2. Make available to the public a watermark decoder that is easy to use by an individual seeking to quickly assess the provenance of a single piece of content, and adheres, to the greatest extent possible, to relevant national or international standards.
   3. Conduct AI red-teaming exercises, as defined, involving third-party experts to test whether watermarks can be easily removed from synthetic content produced by the provider’s GenAI systems, as specified. Red-teaming exercises shall be conducted before the release of any new GenAI system and annually thereafter.
2. Requires a GenAI system capable of producing potentially deceptive content, as defined, to generate and store, in a searchable online database, a digital fingerprint of and provenance data for any piece of potentially deceptive content that they produce, as specified.
3. Prohibits providers and distributors of software and online services from making available a system, application, tool, or service that is designed to remove watermarks from synthetic content.
4. Prohibits GenAI hosting platforms from making available a GenAI system that does not place maximally indelible watermarks communicating provenance data into content created or substantially modified by the system in a manner consistent with specifications pursuant to above.
5. Requires a GenAI provider, within 96 hours of discovering a material vulnerability or failure in a GenAI system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, to report the vulnerability to the California Department of Technology (CDT).
6. Requires a provider to notify other GenAI providers that may be affected by similar vulnerabilities, or failures in a manner that allows the other GenAI provider to harden their own AI systems against similar risks, as specified.
7. Requires a provider to use commercially reasonable efforts to notify affected parties, including, but not limited to, online platforms, researchers or users who received incorrect results from a watermark decoder, or users who produced AI content that contained incorrect or insufficient provenance data, as specified.
8. Requires a provider to make any report to the CDT publicly available in a location linked from the home page of the provider’s internet website with a clearly labeled link that has a similar look, feel, and size relative to other links on the same web page, as specified.
9. Requires a conversational AI system to clearly and prominently disclose to users that the conversational AI system generates synthetic content, as specified.
10. Requires, in all conversational interfaces of a conversational AI system, the conversational AI system to, at the beginning of the user’s interaction with the system, to obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system, as specified.
11. Requires a newly manufactured recording device sold, offered for sale, or distributed in California to offer users the option to place a watermark into content produced by that device, as specified.
12. Requires a recording device manufacturer, if technically feasible, to offer a software or firmware update enabling a user of a recording device manufactured before July 1, 2026, and purchased in California, to do both of the following:
    1. Place a watermark on the content created by the device.
    2. Decode the provenance data.
13. Requires a large online platform to use labels to disclose the provenance data of content distributed on its platform, as specified.
14. Requires the disclosure to be readily legible to an average viewer or, if the content is in audio format, to be clearly audible, as specified.
15. Requires a large online platform to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by GenAI systems without watermarking functionality, as specified.
16. Provides that if content uploaded to or distributed on a large online platform by a user does not contain provenance data, or if the content’s provenance data cannot be interpreted or detected by the platform, the platform must require the user to disclose whether the content is synthetic content, as specified.
17. Requires a large online platform to use state-of-the-art techniques to detect and label text-based potentially deceptive content that is uploaded by users.
18. Requires a large online platform to make accessible some functionality for users to apply a digital signature to non-synthetic content. The functionality shall include options that do not require disclosure of personally identifiable information.
19. Requires a large online platform that can detect potentially deceptive content that does not contain watermarks that comply with applicable industry standards to generate and store, in an online database to be shared and privately accessible by all other online platforms and the CDT, digital fingerprints and any associated provenance data for these images, as specified.
20. Requires GenAI providers, beginning July 1, 2026, and annually thereafter, to produce a Risk Assessment and Mitigation Report that assess the risks posed and harms caused by synthetic content generated by their GenAI systems or hosted on their GenAI hosting platforms.
21. Requires the report to include, but not be limited to, assessments of the distribution of illegal GenAI-generated child sexual abuse materials (CSAM), nonconsensual intimate imagery, disinformation related to elections or public health, plagiarism, or other instances where synthetic or potentially deceptive content caused or may have the potential to cause harm.
22. Requires the report to incorporate information known to the GenAI provider or large online platform about known harms caused by synthetic content generated by their systems or hosted on their platforms, as informed by reports submitted to, and confirmed by, the provider or platform, and independent investigation as appropriate, including, for example, illegal material.
23. Requires the report to be audited by qualified, independent auditors who shall assess and either validate or invalidate the claims made in the report, as specified.
24. Authorizes the CDT to assess an administrative penalty as follows:
    1. If a violation of this bill is intentional or is the result of grossly negligent conduct, a penalty of up to $500,000 for each violation.
    2. If a violation of this bill is not intentional or is not the result of grossly negligent conduct, a penalty of up to $50,000 for each violation.
25. Requires CDT, within 90 days of the date upon which this bill takes effect, to adopt regulations to implement and carry out the purposes of this bill, as specified.
26. Includes definitions of relevant terms, as specified.
27. Provides that the provisions of this bill are severable, as specified.
28. Includes related Legislative findings and declarations.

Background

Author Statement. According to the author’s office:

The author's primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. Specifically, the bill aims to address the interrelated problems stemming from the increasing proliferation and sophistication of generative AI technologies that can create synthetic or "deep fake" content that is difficult to distinguish from human-generated, non-synthetic content:

1. Harms caused by potentially deceptive content presented as human generated content: The bill acknowledges the wide range of potential harms caused by potentially deceptive AI generated content, including financial scams, non-consensual intimate imagery, disinformation (especially around elections and public health), and the erosion of trust in the digital information ecosystem. The PAWS Act seeks to reduce these harms by requiring clear disclosure of content provenance, making it harder for potentially deceptive content to be mistaken as human generated.

2. Lack of transparency around provenance of digital media: The bill addresses transparency concerns by mandating that generative AI providers embed imperceptible and indelible watermarks containing provenance data in all synthetic content they create, and prominently display this provenance data to users. The bill also establishes standards for recording devices to offer watermarking options for human generated content.

3. Facilitation of harmful acts by bad actors: The bill prohibits the distribution of tools designed to remove watermarks or manipulate provenance data, making it more difficult for bad actors to generate unlabeled, potentially deceptive content. It also requires conversational AI systems to disclose their artificial nature and obtain user consent, making it more difficult for bad actors to leverage AI generation for deception.

In sum, the bill seeks to establish clear standards and requirements around content provenance disclosure, watermarking, and labeling, with the goal of increasing transparency and reducing the ability of bad actors to deceive users with unlabeled synthetic content. By doing so, the author aims to mitigate the various potential harms enabled by increasingly sophisticated generative AI technologies.

AI Ascendant: Navigating Novel Technologies and Responsibilities. In the rapidly evolving landscape of technology, AI stands at the forefront, bringing a new era of innovation and application across various sectors. AI, defined broadly, is the capability of machines to perform tasks that typically require human intelligence, encompasses technologies that can process information, learn, reason, recognize patterns, and make decisions.

The transformative potential of AI is vast, affecting disparate industries from healthcare to transportation, and its development is a subject of both enthusiasm and scrutiny. Given the technology's capacity to revolutionize the way we live and work, governments around the world and across the nation are seeking to nurture its growth while addressing the ethical, privacy, and security concerns it raises.

Provenance, in relation to GenAI content, refers to the detailed history and origin of the synthetic content generated by an AI system. This includes information about the creation process, modifications, and the AI system responsible for generating or altering the content. Provenance can ensure transparency and traceability, helping users verify the authenticity of the content and its source.

Metadata, which is data about data, is supplementary data that provides context about the main content. This includes information such as the date and time of creation, the AI model used, version details, and any applied modifications or watermarks. Metadata helps in organizing, identifying, and managing content efficiently, enhancing the overall reliability and accountability of AI-generated material. Image metadata can be viewed using image editing software (like Adobe Photoshop) or through the file properties on a computer. Additionally, media players and editing software often provide access to metadata.

As an example of synthetic content, the image below was created utilizing ChatGPT by simply uploading the text of AB 3211 and asking for an image that best represents the bill:

Virtual Vandalism: When AI Fakes Go Viral. GenAI is a powerful technology capable of creating new content – such as text, images, code, and audio – by learning from existing data. These AI models can produce realistic and innovative artifacts that mimic their training data without directly copying it. For instance, GenAI can craft a poem, create an image, or compose a song based on a prompt or theme. By identifying patterns and structures within existing data through neural networks, GenAI allows users to rapidly generate new content.

The world has marveled at GenAI’s capabilities since the introduction of systems like ChatGPT in late 2022. However, these advancements blur the line between reality and fiction. For example, Detecting AI fingerprints: A guide to watermarking and beyond, a report by the Brookings Institute highlights the issue:

Over the last year, generative AI tools have made the jump from research prototype to commercial product. Generative AI models like OpenAI’s ChatGPT and Google’s Gemini can now generate realistic text and images that are often indistinguishable from human-authored content, with generative AI for audio and video not far behind. Given these advances, it’s no longer surprising to see AI-generated images of public figures go viral or AI-generated reviews and comments on digital platforms. As such, generative AI models are raising concerns about the credibility of digital content and the ease of producing harmful content going forward.

Against the backdrop of such technological advances, civil society and policymakers have taken increasing interest in ways to distinguish AI-generated content from human-authored content.

Further, in an article titled “Experts Say That Soon, Almost The Entire Internet Could Be Generated By AI,” Lonnie Lee Hood quotes Copenhagen Institute for Future Studies (CIFS) expert, Timothy Shoup, stating “that 99 percent to 99.9 percent of the internet’s content will be AI-generated by 2025 to 2030, especially if models like OpenAI’s GPT-3 achieve wider adoption. ‘The internet would be completely unrecognizable,’ Shoup told colleague Sofie Hvitved.”

The potential for misuse is vast, including deep fakes for blackmail or shaming, election interference and misinformation, fraudulent impersonations, and other malicious purposes. For example, in January of this year, Taylor Swift was a victim of widely spread, nonconsensual deep fake images created with GenAI. More alarmingly, a trend has emerged in schools where students use deep fake technology combined with real images to create fake nude photos of classmates. The images can be quickly created using just a smartphone.

Additionally, in February of this year, New Hampshire voters received robocalls that allegedly used an AI-generated voice resembling President Joe Biden, advising them to save their vote for the general election instead of the primary. Recently, a former federal judge urged the federal judiciary’s Advisory Committee on Evidence Rules to update rules on the admissibility of AI-generated evidence. Beyond the concern of AI-generated evidence being admitted is the risk of real evidence being dismissed as synthetic – a phenomenon known as the “liar’s dividend.” As awareness of realistic fakes grows, some will exploit this skepticism to challenge the authenticity of genuine content.

Digital Doppelgangers: Europe’s AI Act and the Quest for Authenticity. Earlier this year, the European Parliament passed the European Union AI Act, addressing the issue of distinguishing between original and AI-generated or manipulated content. The Act, among other things, mandates that developers and users of AI systems help ensure, as much as possible, that people can tell the difference between human-created and AI-generated content. The Act states:

A variety of AI systems can generate large quantities of synthetic content that becomes increasingly hard for humans to distinguish from human-generated and authentic content. The wide availability and increasing capabilities of those systems have a significant impact on the integrity and trust in the information ecosystem, raising new risks of misinformation and manipulation at scale, fraud, impersonation and consumer deception. In light of those impacts, the fast technological pace and the need for new methods and techniques to trace origin of information, it is appropriate to require providers of those systems to embed technical solutions that enable marking in a machine readable format and detection that the output has been generated or manipulated by an AI system and not a human. Such techniques and methods should be sufficiently reliable, interoperable, effective and robust as far as this is technically feasible, taking into account available techniques or a combination of such techniques, such as watermarks, metadata identifications, cryptographic methods for proving provenance and authenticity of content, logging methods, fingerprints or other techniques, as may be appropriate. When implementing this obligation, providers should also take into account the specificities and the limitations of the different types of content and the relevant technological and market developments in the field, as reflected in the generally acknowledged state-of-the-art. Such techniques and methods can be implemented at the level of the system or at the level of the model, including general purpose AI models generating content, thereby facilitating fulfilment of this obligation by the downstream provider of the AI system. To remain proportionate, it is appropriate to envisage that this marking obligation should not cover AI systems performing primarily an assistive function for standard editing or AI systems not substantially altering the input data provided by the deployer or the semantics thereof.

The Act also obligates users of AI systems that generate or manipulate image, audio, or video content to clearly disclose when such content “appreciably resembles existing persons, places, or events and would falsely appear to a person to be authentic (deep fakes), should also clearly and distinguishably disclose that the content has been artificially created or manipulated by labeling the artificial intelligence output accordingly and disclosing its artificial origin.”

There is an ongoing arms race in techniques to distinguish synthetic from authentic content, with many companies committing to identifying such content. Several methods exist for detecting AI-generated or altered content, though none are foolproof and all require continuous updates as technology advances. The Brookings Institute notes that:

There are several approaches that have been proposed for detecting AI-generated content. The four most prominent approaches are watermarking (in its various forms), which is the embedding of an identifiable pattern in a piece of content to track its origin; content provenance, which securely embeds and maintains information about the origin of the content in its metadata; retrieval-based detectors, where all AI-generated content is stored in a database that can be queried to check the origin of content; and post-hoc detectors, which rely on machine learning models to identify subtle but systematic patterns in AI-generated content that distinguish it from human-authored content.

However, a coalition of technology firm associations including TechNet, NetChoice, the Computer & Communications Industry Association, and CalChamber state that:

[C]ontent provenance and watermarking is still incredibly unreliable and in many cases easy to break. Researchers at the University of Maryland were able to break all the currently available watermarking methods. Some can be avoided by simple cropping, resizing, or screenshotting an image. More concerning, these researchers were able to insert fake watermarks and credentials into images, creating false positives. Provenance and watermarking tools tend to help good faith actors act virtuously, but they have limits on stopping bad actors. No provenance solution that’s been created so far, including watermarking or metadata, stops bad actors from simply ‘stripping’ provenance elements and posting a fake piece of content as authentic.

Decoding Deception: Tracking and Labeling AI-Generated Content. This bill places significant responsibility on GenAI providers to ensure traceability of AI-generated or modified content. The bill defines a “GenAI provider” to mean an organization or individual that creates, codes, substantially modifies, or otherwise produces a GenAI system. These providers must embed an imperceptible and “maximally indelible” watermark into synthetic content created or significantly altered by their systems. A “watermark” in this context refers to information embedded in a GenAI system’s output to indicate its synthetic nature, identity, and history, covering images, audio, video, text, or computer code. A “maximally indelible watermark” is designed to be extremely difficult to remove using state-of-the-art techniques and industry standards. “Synthetic content” includes any information, such as images, videos, audio, and text, produced or significantly modified by a GenAI system.

This bill allows for alternative methods of embedding provenance data when the content is too small. Providers are required to design watermarks to the greatest extent possible to convey that the content is synthetic and identify the provider. To ensure consumers can use this information, the bill mandates that GenAI providers offer a freely accessible watermark decoder that is easy to use and meets industry standards. Additionally, providers must conduct structured tests (red-teaming exercises) of their watermarking technology and publicly share the results. GenAI providers must submit a comprehensive report to CDT and notify them of any system vulnerabilities or failures related to the incorrect or malicious inclusion or removal of provenance information or watermarks, along with other reporting obligations.

If their GenAI system can produce “potentially deceptive content,” providers must ensure the system generates and stores a digital fingerprint and provenance data for such content in a searchable online database that can be retrieved by viewers. “Potentially deceptive content” refers to synthetic content that is so similar to non-synthetic content that it could reasonably be mistaken for genuine content.

To reinforce these provisions, the bill prohibits making systems or tools available that are designed to remove watermarks and prohibits GenAI hosting platforms from offering systems that do not comply with the watermarking requirements. This bill also requires conversational AI systems to clearly disclose that they produce synthetic content and obtain the user’s affirmative consent at the start of the interaction.

Tracing Truth: Watermarking for Recording Devices. To complement the marking of synthetic content, this bill also focuses on identifying authentic content. It mandates that recording devices (camera, video recorder, microphone, etc.) sold, offered for sale, or distributed in California to offer users the option to place a watermark into content produced by that device. The watermark must be compatible with industry standards and allow users to customize the type of provenance data included, with the default setting excluding personally identifiable information. Devices must inform users about the watermark settings and indicate when watermarks are being applied.

These requirements will apply to newly manufactured devices, once the bill takes effect on July 1, 2026. Additionally, and if technically feasible, this bill requires manufacturers to offer a software or firmware update for devices made before this date and purchased in California, allowing users to add a watermark to their content and decode the provenance data.

Guardians of the Genuine: Labeling Requirements for Major Online Platforms. This bill assigns several responsibilities to large online platforms, where Californians are most likely to encounter synthetic content. A “large online platform” is defined by the bill to mean a public-facing internet website, web application, or digital application, including a social media platform, video-sharing platform, messaging platform, advertising network, or search engine that had at least one million California users during the preceding 12 months and can facilitate the sharing of synthetic content.

Under this bill, these platforms must label content with its provenance data, clearly identifying whether it is synthetic, non-synthetic, or somewhere in between. The bill mandates the use of state-of-the-art techniques to detect and label synthetic content with removed watermarks or produced by GenAI systems lacking watermarking functionality, as well as potentially deceptive content uploaded by users. If content lacks provenance data, users must disclose if it is synthetic, and it must be labeled as of unknown provenance. Platforms must also provide functionality for users to apply a digital signature to non-synthetic content.

If a large online platform detects potentially deceptive content without industry-standard watermarks, it must generate and store digital fingerprints and associated provenance data in an online database. This database will be shared and will be privately accessible by other online platforms and CDT, excluding personally identifiable information. CDT can share access with coordinating bodies to facilitate faster and more efficient detection and labeling of synthetic content.

Technology industry associations in opposition to the bill argue that, “[a]s a threshold matter, ‘state-of-the-art techniques’ is a term of art used several times throughout the bill and has no legal meaning. It should be struck in favor of a clear standard, especially considering the exorbitant penalties the bill would seek to levy for any violation ($1 million or 5% of global annual revenue, whichever is higher).” It is worth noting that recent amendments taken in the Senate Judiciary Committee reduced those penalties to $500,000 for intentional or “grossly negligent conduct,” and $50,000 for non-intentional or is not the result of “grossly negligent conduct.”

Further, Meta has recently committed to labeling AI-generated images on its platforms. In an article posted to Meta’s internet website in February of this year titled “Labeling AI-Generated Images on Facebook, Instagram and Threads,” Meta states that it is committed to “label images that users post to Facebook, Instagram and Threads when we can detect industry standard indicators that they are AI-generated.” A group of technology companies including Adobe, Google, and Microsoft, has established the Coalition for Content Provenance and Authenticity (C2PA) to address “the prevalence of misleading information online through the development of technical standards for certifying the source and history (or provenance) of media content.” OpenAI announced that it will add C2PA metadata to images created within ChatGPT and the API for the DALL-E 3 model.

Annual Risk and Harm Assessments. Beginning July 1, 2026, and annually thereafter, this bill mandates that GenAI providers and large online platforms produce a Risk Assessment and Mitigation Report. These reports must evaluate the risks and harms associated with synthetic content generated by their GenAI systems or hosted on their platforms. The reports need to cover various specified areas, including the illegal distribution of GenAI CSAM and non-consensual intimate imagery (or “deep fake” porn). Additionally, the reports must detail known harms caused by synthetic content from their systems or platforms. Independent audits are required to validate the claims made in these reports.

Enforcement. This bill authorizes CDT to assess administrative penalties of up to $500,000 for each violation that is intentional or is the result of grossly negligent conduct, and up to $50,000 for each violation that is not intentional or is not the result of grossly negligent conduct.

Additionally, this bill requires CDT to adopt regulations to implement and carry out the purposes of this bill within 90 days of the date upon which this bill takes effect. CDT will review and update its regulations relating to the implementation of this bill as needed, including, but not limited to, adopting specific national or international standards for provenance, authenticity, watermarking, and digital signatures, as long as the standards do not weaken the provisions of the bill.

Regulatory Rulemaking Process. The Office of Administrative Law (OAL) is responsible for ensuring that California state agencies comply with the rulemaking procedures and standards set forth in California’s APA. A “regulation” is any rule, regulation, order or standard of general application or the amendment, supplement, or revision of any rule, regulation, order, or standard adopted by any state agency to implement, interpret, or make specific the law enforced or administered by it. When adopting regulations, every department, division, office, officer, bureau, board or commission in the executive branch of the California state government must follow the rulemaking procedures in the APA (Government Code section 11340 et seq.) and regulations adopted by the, unless expressly exempted by statute from some or all of these requirements.

The APA requirements are designed to provide the public with a meaningful opportunity to participate in the adoption of regulations or rules that have the force of law by California state agencies and to ensure the creation of an adequate record for the OAL and judicial review.

Additional Action at a Distance: Federal AI Policies. Similar legislation, the AI Labeling Act of 2023, has been introduced at the federal level by Senators Schatz and Kennedy. Additionally, President Biden issued an executive order last year on AI, setting standards for safety and security to ensure responsible AI development and deployment in the United States. The order defines GenAI as “the class of AI models that emulate the structure and characteristics of input data to generate derived synthetic content, including images, videos, audio, text, and other digital content.”

The executive order calls for the development of a risk management framework for GenAI, modeled after the AI Risk Management Framework created by the National Institute for Standards and Technology (NIST).

The Three Bill Problem. This bill is one dozens of bills introduced this Legislative Session attempting to address the emerging technologies that encompass AI. Specifically, this is one of three bills focused on GenAI content provenance. Earlier this year, this Committee heard and approved SB 942 (Becker, 2024) – the AI Transparency Act – which requires the developers of GenAI systems to both include provenance disclosures in the content their systems produce, and make tools available to identify GenAI-content produced by their systems. That bill is pending in the Assembly Judiciary Committee.

Additionally, AB 1791 (Weber, 2024) requires social media platforms to redact “personal provenance data” from content uploaded on their platforms while retaining “system provenance data” that reveals information about the creation of the content. The bill requires platforms to redact both if inseparable, and to append a label with the latter data. That bill is pending in the Senate Appropriations Committee.

Prior/Related Legislation

SB 896 (Dodd, 2024) the GenAI Accountability Act establishes guidelines for the use and regulation of GenAI within state government operations, as specified, and requires a state agency utilizing GenAI to directly communicate with a person to clearly and in a conspicuous way identify to that person that the communication is being facilitated by GenAI, as specified. (Pending in the Assembly Privacy and Consumer Protection Committee)

SB 942 (Becker, 2024) the AI Transparency Act requires the developers of GenAI systems to both include provenance disclosures in the content their systems produce, and make tools available to identify GenAI-content produced by their systems. (Pending in the Assembly Judiciary Committee)

SB 970 (Ashby, 2024) would have ensured that media manipulated or generated by AI technology is incorporated into the right of publicity law and criminal false impersonation statutes, as specified. (Held on the Senate Appropriations Committee Suspense File)

AB 1791 (Weber, 2024) requires social media platforms to redact “personal provenance data” from content uploaded on their platforms while retaining “system provenance data” that reveals information about the creation of the content. The bill requires platforms to redact both if inseparable, and to append a label with the latter data. (Pending in the Senate Appropriations Committee)

AB 2930 (Bauer-Kahan, 2024) among other things, requires, among other things, a deployer and a developer of an automated decision tool to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts, as specified. (Pending in the Senate Judiciary Committee)

SB 313 (Dodd, 2023) the California AI-ware Act would have established, within CDT, the Office of AI, as specified, and would have required a state agency utilizing GenAI to directly communicate with a person to clearly and in a conspicuous way identify to that person that the communication is being facilitated by GenAI, as specified. (Held on the Senate Appropriations Committee Suspense File)

SCR 17 (Dodd, Res. Chapter 135, Statutes of 2023) affirms the California Legislature’s commitment to President Biden’s vision for a safe AI and the principles outlined in the “Blueprint for an AI Bill of Rights,” and expresses the Legislature’s commitment to examining and implementing those principles in its legislation and policies related to the use and deployment of automated systems.

SB 1216 (Gonzalez, Chapter 885, Statues of 2022) requires, upon appropriation by the Legislature, the Secretary of GovOps to evaluate, among other things, the impact that the proliferation of deep fakes has on government, businesses, and residents of the state, as specified.

AB 1576 (Calderon, 2019) would have required the Secretary of GovOps to appoint participants to an AI working group to evaluate the uses, risks, benefits, and legal implications associated with the development and deployment of AI by California-based businesses. (Held on the Senate Appropriations Committee Suspense File)

SB 1001 (Hertzberg, Chapter 892, Statutes of 2018) makes it unlawful for any person to use a bot to communicate or interact with another person online with the intent to mislead the other person about its artificial identity for the purpose of knowingly deceiving the person about the content of the communication in order to incentivize a purchase or sale of goods or services in a commercial transaction or to influence a vote in an election, as specified.

FISCAL EFFECT: Appropriation: Fiscal Com.: Local:

SUPPORT:

California Initiative for Technology & Democracy (Source)

Accountable Tech
AFSCME, AFL-CIO
Asian Americans and Pacific Islanders for Civic Empowerment
Asian Law Alliance
Bay Rising
Board of Supervisors for The City and County of San Francisco
California Clean Money Campaign
California Voter Foundation
Catalyst California
Center for Countering Digital Hate
Check My Ads
Chinese Progressive Association
Courage California
Hmong Innovating Politics
MOVE the Valley
NextGen California
Partnership for the Advancement of New Americans
SEIU California
TechEquity Action
Young People's Alliance

OPPOSITION:

California Chamber of Commerce

Computer & Communications Industry Association

NetChoice

Oakland Privacy

TechNet

ARGUMENTS IN SUPPORT: In support of the bill, the California Initiative for Technology & Democracy writes that, “[t]his bill makes critical interventions at two stages of our information system, first at the point of content creation and then at the point of dissemination. At the point of content creation, AB 3211 would enable human-created authentic content to be differentiated from AI-generated synthetic content by requiring separate watermarking standards for each. AB 3211 would require cameras and recording devices sold in California to include an option to place an authenticity watermark on the content that the device produces. Separately, AI companies would be required to place imperceptible and maximally-indelible watermarks on any content that their AI systems produce. These watermarks would contain provenance information, allowing users, social media platforms, and the public to readily trace the source of the content. AB 3211 would also impose rigorous testing and reporting standards, as well as impact assessments, to ensure watermarks are resistant to manipulation and removal.”

Thereafter, “at the point of content dissemination, AB 3211 would require social media and other online platforms to adopt state-of-the-art techniques to identify AI-generated content posted on their platform, including leveraging the provenance data contained in watermarked content. Once the source has been differentiated, platforms would then be required to label content as ‘fully synthetic,’ ‘partially synthetic,’ ‘authentic,’ ‘authentic with minor modifications to color, noise reduction, etc.,’ or ‘does not contain a watermark.’ By requiring clear, factual labeling of the source of online content, especially AI-generated content, AB 3211 would equip the public with a tool to make their own judgement about what information they deem to be trustworthy.”

ARGUMENTS IN OPPOSITION: A coalition of industry associations write, in part, that “[m]any of our companies and platforms are at the forefront of developing content provenance and watermarking technology, which is still in its early stages. However, AB 3211 enacts incredibly prescriptive requirements for a technology that is still under development and rapidly evolving. For example, watermarking text is still in its infancy, making the bill’s requirements to do so nearly impossible to comply with.”

Furthermore, “[i]n its standards for large online platforms, AB 3211 should more clearly delineate between 1st party and 3rd party content. 1st party content would be images, videos, or audio that is generated using a large online platform’s generative AI tools and is then posted or distributed on that platform. In this instance, a platform can actually control the creation of a content provenance or watermark into the content. As mentioned, many of our companies are already working to incorporate this type of technology to increase transparency around AI-generated content. It is currently technically infeasible to accurately and reliably detect content that is created using a different platform’s AI tools. As noted above, considering the current ease with which current watermarks can be broken, a legal requirement and mandate for 3rd party content isn’t appropriate.”

Additionally, “AB 3211 seems to treat all AI-generated content as inherently bad or risky. By requiring such thorough and prescriptive requirements for content labeling, the bill makes a value judgement that consumers must be notified and aware of any content that was created by AI. This applies to all inauthentic content including purely artistic or satirical content. While we agree with the intent to provide more information to consumers, in some instances it could create disclosure or notification fatigue. If watermarks and content credentials become so routine and placed on all AI-generated content, users may start to ignore and disregard their presence. Rather than focusing on whether the content itself was AI-generated, synthetic, or inauthentic, we would advise focusing on the misuse of this technology.”

DUAL REFERRAL: Senate Judiciary Committee (8-2) & Senate Governmental Organization Committee

SENATE JUDICIARY COMMITTEE

Senator Thomas Umberg, Chair

2023-2024 Regular Session

Bill No: AB 3211 (Author:Wicks)

Version: June 10, 2024

Hearing Date: June 18, 2024

Fiscal: Yes

Urgency: No

Consultant: CK

SUBJECT

California Provenance, Authenticity and Watermarking Standards

DIGEST

This bill establishes the California Provenance, Authenticity and Watermarking Standards Act, which requires a generative AI (GenAI) system provider to, among other things, take certain actions to assist in the disclosure of provenance data to mitigate harms caused by inauthentic content, including placing imperceptible and maximally indelible watermarks containing provenance data into content created by its systems. The bill requires a large online platform, as defined, to, among other things, use labels to prominently disclose the provenance data found in watermarks or digital signatures in content distributed to users on its platforms, as specified, and to use state-of-the-art techniques to detect and label inauthentic text content that is uploaded or distributed by users. The bill requires GenAI providers and platforms to produce annual impact assessment reports. The bill also requires recording device manufacturers to enable options for embedding provenance data into recordings.

EXECUTIVE SUMMARY

Certain forms of media – audio recordings, video recordings, and still images – can be powerful evidence of the truth. While such media have always been susceptible to some degree of manipulation, fakes were relatively easy to detect. The rapid advancement of AI technology, specifically the wide-scale introduction of GenAI models, has made it drastically cheaper and easier to produce synthetic content, audio, images, text, and video recordings that are not real, but that are so realistic that they are virtually impossible to distinguish from authentic content, including so-called “deepfakes.”

Among other things, this bill works to ensure that providers of these GenAI systems are equipping consumers with a tool to identify when specific content has been generated by their systems, places requirements on large online platforms to disclose the provenance data of content and to detect and label inauthentic content, and requires recording device manufacturers to allow for the embedding of watermarks on recordings. This bill is sponsored by the California Initiative for Technology & Democracy and supported by a wide variety of groups, including SEIU California and NextGen CA. The bill is opposed by Oakland Privacy and a coalition of industry groups, including Netchoice. Should this pass out of this Committee, it will next be heard in the Senate Governmental Organization Committee.

PROPOSED CHANGES TO THE LAW

Existing law:

1. Defines “deepfake” as audio or visual content that has been generated or manipulated by artificial intelligence which would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do without their consent. (Gov't Code § 11547.5.)
2. Defines “digital content forgery” as the use of technologies, including artificial intelligence and machine learning techniques, to fabricate or manipulate audio, visual, or text content with the intent to mislead. (Gov't Code § 11547.5.)
3. Defines “digital content provenance” as the verifiable chronology of the original piece of digital content, such as an image, video, audio recording, or electronic document. (Gov't Code § 11547.5.)
4. Requires, upon appropriation by the Legislature, the Secretary of Government Operations Agency (GovOps) to evaluate the following:
   1. The impact of the proliferation of deepfakes on state government, California-based businesses, and residents of the state.
   2. The risks, including privacy risks, associated with the deployment of digital content forgery technologies and deepfakes on state and local government, California-based businesses, and residents of the state.
   3. Potential privacy impacts of technologies allowing public verification of digital content provenance.
   4. The impact of digital content forgery technologies and deepfakes on civic engagement, including voters.
   5. The legal implications associated with the use of digital content forgery technologies, deepfakes, and technologies allowing public verification of digital content provenance.
   6. The best practices for preventing digital content forgery and deepfake technology to benefit the state, California-based businesses, and California residents, including exploring whether and how the adoption of a digital content provenance standard could assist with reducing the proliferation of digital content forgeries and deepfakes. (Gov’t Code § 11547.5(b).)
5. Requires the Secretary of GovOps to develop a coordinated plan to accomplish all of the following:
   1. Investigate the feasibility of, and obstacles to, developing standards and technologies for state departments for determining digital content provenance.
   2. Increase the ability of internet companies, journalists, watchdog organizations, other relevant entities, and members of the public to meaningfully scrutinize and identify digital content forgeries and relay trust and information about digital content provenance to content consumers.
   3. Develop or identify mechanisms for content creators to cryptographically certify authenticity of original media and nondeceptive manipulations.
   4. Develop or identify mechanisms for content creators to enable the public to validate the authenticity of original media and nondeceptive manipulations to establish digital content provenance without materially compromising personal privacy or civil liberties. (Gov’t Code § 11547.5(c).)
6. Establishes the California Department of Technology (CDT) within GovOps, under the supervision of the Director of Technology (Director), also known as the State Chief Information Officer. (Gov. Code Sec. 11545(a).)
7. Provides that the duties of the Director include:
   1. Advising the Governor on the strategic management and direction of the state’s information technology (IT) resources.
   2. Establishing and enforcing state IT strategic plans, policies, standards, and enterprise architecture, as specified.
   3. Minimizing overlap, redundancy, and cost in state IT operations by promoting the efficient and effective use of information technology.
   4. Providing technology direction to agency and department chief information officers to ensure the integration of statewide technology initiatives, compliance with IT policies and standards, and the promotion of the alignment and effective management of IT services.
   5. Working to improve organizational maturity and capacity in the effective management of IT; and establishing performance management and improvement processes to ensure state IT systems and services are efficient and effective. (Gov. Code § 11545(b).)

This bill:

1. Defines the relevant terms, including:
   1. “Digital fingerprint” means a unique value that can be used to identify identical or similar digital content.
   2. “Digital signature” means a method based on cryptography that allows a user or entity to digitally sign content with provenance data in order to verify that the user or entity participated in the creation of the content.
   3. “Generative AI system” means an artificial intelligence system that generates derived synthetic content, including images, videos, audio, text, and other digital content.
   4. “Large online platform” means a public-facing internet website, web application, or digital application, including a social network, video-sharing platform, messaging platform, advertising network, or search engine that had at least 1,000,000 California users during the preceding 12 months and can facilitate the sharing of synthetic content.
   5. “Maximally indelible watermark” means a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant industry standards.
   6. ”Nonsynthetic content” means images, videos, audio, or text created by human beings without any modifications or with only minor modifications that do not lead to significant changes to the perceived content or meaning of the content. Minor modifications include, but are not limited to, changes to brightness or contrast of images, removal of background noise in audio, and spelling or grammar corrections in text.
   7. “Potentially deceptive content” means synthetic content that is so similar to nonsynthetic content that it could reasonably be mistaken as nonsynthetic content.
   8. “Provenance data” means information about the history of the content, including, but not limited to, the following:
      1. The name of the generative AI provider or the camera or recording device manufacturer.
      2. The name and version number of the AI system that generated the content or the operating system, version of the operating system, or the application used to capture, create, or record the content.
      3. The time and date of the content’s creation and any additional modifications of the content.
      4. The portions of content that have been changed by a generative AI system, if applicable.
   9. “Synthetic content” means information, including images, videos, audio, and text, that has been produced or significantly modified by a generative AI system.
   10. “Watermark” means information that is embedded into content for the purpose of communicating the provenance, history of modification, or history of conveyance.
2. Requires a GenAI provider to place an imperceptible and maximally indelible watermark into synthetic content produced or significantly modified by the provider’s GenAI system, as provided. To the greatest extent possible, watermarks shall be designed to communicate information that identifies content as synthetic and identifies the provider in the event that a sample of synthetic content is corrupted, downscaled, cropped, or otherwise damaged.
3. Requires a GenAI provider to make available to the public a watermark decoder that is easy to use and adheres to relevant national or international standards.
4. Requires a GenAI provider to conduct AI red-teaming exercises involving third-party experts to test whether watermarks can be easily removed or whether the provider’s GenAI systems can be used to falsely add watermarks to otherwise nonsynthetic content. Red-teaming exercises shall be conducted before the release of any new Gen AI system and annually thereafter, as provided. A provider shall make summaries of these exercises publicly available, as specified, and full reports are to be sent to CDT. Details that pose an immediate risk to public safety or provide information that could be used to disable or circumvent the functionality of watermarks must be removed.
5. Requires a GenAI system capable of producing potentially deceptive content to generate and store, in a searchable online database in a manner that can be retrieved by a viewer of the content, a digital fingerprint of and provenance data for any piece of potentially deceptive content that they produce. This provenance shall not include personally identifiable information.
6. Prohibits providers and distributors of software and online services from making available a system, application, tool, or service that is designed to remove watermarks from synthetic content.
7. Prohibits a GenAI hosting platform from making available a GenAI system that does not place maximally indelible watermarks, as required herein.
8. Requires a GenAI provider to, within 96 hours of discovering a material vulnerability or failure in a GenAI system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, report the vulnerability or failure to CDT, as provided. Providers must also notify other providers that may be affected by similar vulnerabilities or failures.
9. Requires a provider to notify other GenAI providers that may be affected by similar vulnerabilities or failures in a manner that allows the other provider to harden their own AI systems against similar risks, but that does not compromise the reporting provider’s systems or disclose the reporting provider’s confidential or proprietary information. A provider must also use commercially reasonable efforts to notify affected parties, including, but not limited to, online platforms, researchers or users who received incorrect results from a watermark decoder, or users who produced AI content that contained incorrect or insufficient provenance data. A provider shall not be required to notify an affected party whose contact information the provider has not previously collected or retained.
10. Requires a conversational AI system to clearly and prominently disclose to users that the conversational AI system generates synthetic content, as specified. In all conversational interfaces of such a system, the system shall, at the beginning of a user’s interaction, obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system before beginning the conversation, as specified.
11. Makes the above section operative on February 1, 2025.
12. Requires, beginning January 1, 2026, newly manufactured recording devices sold, offered for sale, or distributed in California to offer users the option to place a watermark into content produced by that device. A user shall have the option to customize the types of provenance data communicated by these watermarks, including by removing any personally identifiable information (PII). PII cannot be included by default.
13. Requires the recording devices to clearly inform users of the existence of the watermark settings upon a user’s first use of the recording function and to contain a clear indicator that a watermark is being applied. A watermark shall, if enabled, be applied to nonsynthetic content produced using third-party applications that bypass default recording applications in order to offer recording functionalities and must be compatible with widely used industry standards.
14. Requires, beginning January 1, 2026, if technically feasible, a recording device manufacturer to offer a software or firmware update enabling a user of a recording device manufactured before January 1, 2026, and purchased in California to place a watermark on the content created by the device and to decode the provenance data.
15. Requires a large online platform, beginning March 1, 2025, to use labels to disclose the provenance data of content distributed on its platform, as specified. The labels shall prominently display whether content is fully synthetic, partially synthetic, nonsynthetic, nonsynthetic with minor modifications, or does not contain a watermark. A user shall be able to click or tap on a label to inspect provenance data in an easy-to-understand format.
16. Requires a large online platform to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by GenAI systems without watermarking functionality. If the platform is not able to detect the provenance data of content, then the platform shall label the content as unknown provenance.
17. Provides that if content uploaded to or distributed on a platform does not contain provenance data, or if the content’s provenance data cannot be interpreted or detected by the platform, a platform shall require the user to disclose whether the content is synthetic content or if they are uncertain. A large online platform shall include prominent warnings to users that uploading or distributing synthetic content without disclosing that it is synthetic content may be a violation of platform policy.
18. Requires a large online platform to use state-of-the-art techniques to detect and label text-based potentially deceptive content that is uploaded by users.
19. Requires a large online platform to make accessible some functionality for users to apply a digital signature to nonsynthetic content, and include options that do not require disclosure of PII.
20. Requires a large online platform that can detect potentially deceptive content that does not contain watermarks that comply with applicable industry standards to generate and store, in an online database to be shared and made privately accessible by all other online platforms and CDT, digital fingerprints and any associated provenance data for these images. This provenance data shall not include PII.
21. Authorizes CDT to share access to these databases with coordinating bodies acting to facilitate more rapid and computationally efficient detection and labeling of synthetic content.
22. Requires GenAI providers and large online platforms, beginning January 1, 2026, and annually thereafter, to produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated by their GenAI systems or hosted on their GenAI hosting platforms. This must include assessments of the distribution of illegal GenAI-generated child sexual abuse materials, nonconsensual intimate imagery, disinformation related to elections or public health, plagiarism, or other instances where synthetic or potentially deceptive content caused or may have the potential to cause harm.
23. Requires the report to incorporate information known to the GenAI provider or large online platform about known harms caused by synthetic content generated by their systems or hosted on their platforms, as informed by reports submitted to, and confirmed by, the provider or platform, and independent investigation as appropriate, including, for example, illegal material. The report must be audited by qualified, independent auditors who shall assess and either validate or invalidate the claims made in the report. Auditors shall use state-of-the-art techniques to assess reports, and shall adhere to relevant national and international standards.
24. Provides that a violation of these provisions may result in an administrative penalty, assessed by CDT, of up to $1 million or five percent of the violator’s annual global revenue, whichever is greater.
25. Requires CDT, within 90 days of the effective date of this bill, to adopt regulations to implement and carry out the purposes of the bill. CDT is required to review and update its regulations as needed, including adopting specific national or international standards for provenance, authenticity, watermarking, and digital signatures, as long as the standards do not weaken these provisions.
26. Includes a severability clause.

COMMENTS

1. Blurring reality: AI-generated content

Generative AI is a type of artificial intelligence that can create new content, including text, images, code, or music, by learning from existing data. Generative AI models can produce realistic and novel artifacts that resemble the data they were trained on, but do not copy it. For example, generative AI can write a poem, draw a picture, or compose a song based on a given prompt or theme. Generative AI enables users to quickly generate new content based on a variety of inputs. Generative AI models use neural networks to identify the patterns and structures within existing data to generate new and original content.

The world has been in awe of the powers of this generative AI since the widespread introduction of AI systems such as the various iterations of ChatGPT. However, the capabilities of these advanced systems leads to a blurring between reality and fiction. The Brookings Institution lays out the issue:

Over the last year, generative AI tools have made the jump from research prototype to commercial product. Generative AI models like OpenAI’s ChatGPT and Google’s Gemini can now generate realistic text and images that are often indistinguishable from human-authored content, with generative AI for audio and video not far behind. Given these advances, it’s no longer surprising to see AI-generated images of public figures go viral or AI-generated reviews and comments on digital platforms. As such, generative AI models are raising concerns about the credibility of digital content and the ease of producing harmful content going forward.

Against the backdrop of such technological advances, civil society and policymakers have taken increasing interest in ways to distinguish AI-generated content from human-authored content.^[1]

One expert at the Copenhagen Institute for Future Studies estimates that should large generative-AI models run amok, up to 99 percent of the internet’s content could be AI-generated by 2025 to 2030.^[2] The problematic applications are seemingly infinite, whether it be deepfakes to blackmail or shame victims, misinformation in elections, false impersonations to commit fraud, or other nefarious purposes. Infamously, in January of this year, Taylor Swift was the victim of sexually explicit, nonconsensual deepfake images using AI that were widely spread across social media platforms.^[3] Perhaps more disturbingly, a trend has emerged in schools of students creating such images: “At schools across the country, people have used deepfake technology combined with real images of female students to create fraudulent images of nude bodies. The deepfake images can be produced using a cellphone.”^[4]

In February of this year, voters in New Hampshire received robocalls that are purported to have used an AI voice resembling President Joe Biden advising them against voting in the presidential primary and saving their vote for the November general election.^[5] Recently, a former federal judge urged the federal judiciary’s Advisory Committee on Evidence Rules to update evidentiary rules regarding the admissibility of evidence believed to be AI generated.^[6] But, in addition to concerns about the potential for AI-generated evidence to be admitted is the reverse, false claims that real evidence is synthetic. As more of the population becomes aware of the potential to realistically fake images, video, and text, some will use the skepticism that creates to challenge the authenticity of real content, a phenomena coined the “liar’s dividend.”^[7]

1. Taking action to identify synthetic content and address its usage

Earlier this year, the European Parliament signed the European Union AI Act. It highlights these very issues and obligates developers and deployers to assist in ensuring, to the extent feasible, that individuals are able to distinguish between original and AI-generated or manipulated content. The Act states:

A variety of AI systems can generate large quantities of synthetic content that becomes increasingly hard for humans to distinguish from human-generated and authentic content. The wide availability and increasing capabilities of those systems have a significant impact on the integrity and trust in the information ecosystem, raising new risks of misinformation and manipulation at scale, fraud, impersonation and consumer deception. In light of those impacts, the fast technological pace and the need for new methods and techniques to trace origin of information, it is appropriate to require providers of those systems to embed technical solutions that enable marking in a machine readable format and detection that the output has been generated or manipulated by an AI system and not a human. Such techniques and methods should be sufficiently reliable, interoperable, effective and robust as far as this is technically feasible, taking into account available techniques or a combination of such techniques, such as watermarks, metadata identifications, cryptographic methods for proving provenance and authenticity of content, logging methods, fingerprints or other techniques, as may be appropriate. When implementing this obligation, providers should also take into account the specificities and the limitations of the different types of content and the relevant technological and market developments in the field, as reflected in the generally acknowledged state-of-the-art. Such techniques and methods can be implemented at the level of the system or at the level of the model, including general purpose AI models generating content, thereby facilitating fulfilment of this obligation by the downstream provider of the AI system. To remain proportionate, it is appropriate to envisage that this marking obligation should not cover AI systems performing primarily an assistive function for standard editing or AI systems not substantially altering the input data provided by the deployer or the semantics thereof.

It also specifically obligates deployers who use an AI system to generate or manipulate image, audio, or video content that “appreciably resembles existing persons, places or events and would falsely appear to a person to be authentic (deep fakes), should also clearly and distinguishably disclose that the content has been artificially created or manipulated by labelling the artificial intelligence output accordingly and disclosing its artificial origin.”

There is currently an arms race in techniques for distinguishing between synthetic and authentic content and companies are declaring their commitment to identifying such content. There are various methods for deciphering AI-generated or altered content, although none are foolproof and all require updates as technology advances:

There are several approaches that have been proposed for detecting AI-generated content. The four most prominent approaches are watermarking (in its various forms), which is the embedding of an identifiable pattern in a piece of content to track its origin; content provenance, which securely embeds and maintains information about the origin of the content in its metadata; retrieval-based detectors, where all AI-generated content is stored in a database that can be queried to check the origin of content; and post-hoc detectors, which rely on machine learning models to identify subtle but systematic patterns in AI-generated content that distinguish it from human-authored content.^[8]

1. Ensuring sufficient tools to establish the provenance of synthetic and nonsynthetic content

This bill looks to implement some of these approaches to ensuring individuals can identify when content is modified by GenAI. It places a wide set of obligations on GenAI providers, recording device manufacturers, and large online platforms.

According to the author:

The primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. Specifically, the bill aims to address the interrelated problems stemming from the increasing proliferation and sophistication of generative AI technologies that can create synthetic or "deepfake" content that is difficult to distinguish from human-generated, non-synthetic content:

• • 1. Harms caused by potentially deceptive content presented as human generated content: The bill acknowledges the wide range of potential harms caused by potentially deceptive AI generated content, including financial scams, non-consensual intimate imagery, disinformation (especially around elections and public health), and the erosion of trust in the digital information ecosystem. The PAWS Act seeks to reduce these harms by requiring clear disclosure of content provenance, making it harder for potentially deceptive content to be mistaken as human generated.
    2. Lack of transparency around provenance of digital media: The bill addresses transparency concerns by mandating that generative AI providers embed imperceptible and indelible watermarks containing provenance data in all synthetic content they create, and prominently display this provenance data to users. The bill also establishes standards for recording devices to offer watermarking options for human generated content.
    3. Facilitation of harmful acts by bad actors: The bill prohibits the distribution of tools designed to remove watermarks or manipulate provenance data, making it more difficult for bad actors to generate unlabeled, potentially deceptive content. It also requires conversational AI systems to disclose their artificial nature and obtain user consent, making it more difficult for bad actors to leverage AI generation for deception.

In sum, the bill seeks to establish clear standards and requirements around content provenance disclosure, watermarking, and labeling, with the goal of increasing transparency and reducing the ability of bad actors to deceive users with unlabeled synthetic content. By doing so, [I aim] to mitigate the various potential harms enabled by increasingly sophisticated generative AI technologies.

• 1. GenAI providers: tracking synthetic content

The bill places a strong obligation on GenAI providers to create the trail for tracking what is GenAI produced or modified content. Providers are required to place an imperceptible and “maximally indelible” watermark into synthetic content produced or significantly modified by their systems. “Watermark,” for these purposes, is information that is embedded into a GenAI system’s output for the purpose of conveying its synthetic nature, identity, content, including image, audio, video, text, or computer code, for the purpose of communicating the provenance, history of modifications, modification, or history of conveyance. “Maximally indelible watermark” is defined as a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant industry standards. “Synthetic content” simply means information, including images, videos, audio, and text, that has been produced or significantly modified by a generative AI system.

The bill provides for alternative methods of embedding provenance data where the content is too small. Ultimately, the bill requires providers to design watermarks, to the greatest extent possible, to communicate information that identifies the content as synthetic and identifies the provider. To make this information useful for consumers, the bill further requires GenAI providers to make a watermark decoder freely available to the public that is easy to use and adheres to relevant industry standards. The providers are also required to conduct red-teaming exercises, essentially structured testing of their watermarking technology, and to make the results public. GenAI providers must provide a full report to CDT and notify them of any identified vulnerabilities or failures in the system related to the erroneous or malicious inclusion or removal of provenance information or watermarks, along with other reporting requirements.

If their GenAI system is capable of producing “potentially deceptive content,” providers must ensure the systems can generate and store, in a searchable online

database in a manner that can be retrieved by a viewer of the content, a digital fingerprint of and provenance data^[9] for any piece of such content that they produce. “Potentially deceptive content” means synthetic content that is so similar to nonsynthetic content that it could reasonably be mistaken as nonsynthetic content.

To ensure wider protections for this approach, the bill prohibits making available systems or tools that are designed to remove watermarks and GenAI hosting platforms from making available systems that are not compliant with the watermarking requirements laid out above.

The bill also requires conversational AI systems to clearly and prominently disclose that they produce synthetic content, as provided. At the outset of a user’s interaction, such systems must obtain the user’s affirmative consent acknowledging the user has been so informed.

The above provisions become operative on February 1, 2025.

• 1. Recording device manufacturers and provenance data

On the flip side of ensuring that synthetic content is properly marked to decipher it, is establishing the ability to identify authentic content. This bill requires recording devices sold, offered for sale, or distributed in California to offer users the option to place a watermark, compatible with industry standards, into content produced by that device. Users would have the option to customize the type of provenance data that is so included, but personally identifiable information must be excluded by default. The devices are required to inform users of the watermark settings and shall indicate when watermarks are being applied.

These requirements take effect on January 1, 2026 and apply to newly manufactured devices. However, the bill requires manufacturers, if technically feasible, to offer a software or firmware update enabling a user of a recording device made before that date and purchased in California, to place a watermark on the content it creates and to decode the provenance data.

• 1. Large online platforms

The bill also includes a number of responsibilities for online platforms, where Californians are most likely to interact with synthetic content. It defines a “large online platform” as a public-facing internet website, web application, or digital application, including a social network,^[10] video-sharing platform, messaging platform, advertising network, or search engine that had at least 1,000,000 California users during the preceding 12 months and can facilitate the sharing of synthetic content.

The platforms are required to label content distributed thereupon disclosing its provenance data. The bill places a number of requirements on the labels, including that they must clearly identify whether content is synthetic, nonsynthetic or somewhere in between.

Platforms are required to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by GenAI systems without watermarking functionality and potentially deceptive content uploaded by users.

If content does not include provenance data, users must be required to disclose whether it is synthetic and it must be labeled as of unknown provenance. Users must be provided some functionality for users to apply a digital signature to nonsynthetic content.

A large online platform that can detect potentially deceptive content that does not contain watermarks that comply with applicable industry standards shall generate and store, in an online database to be shared and privately accessible by all other online platforms and CDT, digital fingerprints and any associated provenance data for these images, excluding personally identifiable information. CDT is authorized to share access to these databases with coordinating bodies acting to facilitate more rapid and computationally efficient detection and labeling of synthetic content.

These requirements take effect March 1, 2025.

• 1. Impact assessments

The bill further requires, beginning January 1, 2026, and annually thereafter, GenAI providers and large online platforms to produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated

by their GenAI systems or hosted on their GenAI hosting platforms. These reports are required to include a host of specified information, including assessments of the illegal distribution of GenAI child sexual abuse material and nonconsensual intimate imagery (or “deepfake” porn). It shall also include information about known harms caused by synthetic content generated by their systems or hosted on their platforms. Audits of these reports by independent entities are also required to validate the claims made.

• 1. Enforcement

CDT is authorized to assess massive administrative penalties of up to $1 million or five percent of the violator’s annual global revenue, whichever is greater, for every violation. CDT is given 90 days from the effective date of the bill to promulgate regulations to implement and carry out the law.

1. Concerns with the bill

A number of stakeholders have raised issues with the sheer breadth of the bill and the

inability to sift through its many provisions in time for this Committee’s hearing, especially given the significant amendments recently taken. Specific to the substance, concerns have also been raised in response to this and other measures regarding the feasibility of watermarking and other digital provenance requirements given the nascent stage of their development. Many existing tools suffer from inaccuracies and false positives. Various technical methods for synthetic content transparency, such as watermarking, fingerprinting, or metadata, are being developed and are evolving but are far from foolproof, as individuals can, for instance, sometimes find ways to remove or obscure these disclosures from content.

A coalition in opposition, including TechNet and NetChoice, argue the bill’s “prescriptive requirements on content provenance and watermarking are technologically premature.” They state:

Many of our companies and platforms are at the forefront of developing content provenance and watermarking technology, which is still in its early stages. However, AB 3211 enacts incredibly prescriptive requirements for a technology that is still under development and rapidly evolving. For example, there isn’t a program that can watermark text, making the bill’s requirements to do so impossible to comply with. We believe references to text watermarking should be removed to reflect this reality.

Furthermore, content provenance and watermarking is still incredibly unreliable and in many cases easy to break. Researchers at the University of Maryland were able to break all the currently available watermarking methods. Some can be avoided by simple cropping, resizing, or screenshotting an image. More concerning, these researchers were able to insert fake watermarks and credentials into images, creating false positives. Provenance and watermarking tools tend to help good faith actors act virtuously, but they have limits on stopping bad actors. No provenance solution that’s been created so far, including watermarking or metadata, stops bad actors from simply ‘stripping’ provenance elements and posting a fake piece of content as authentic.

In its standards for large online platforms, AB 3211 should more clearly delineate between 1st party and 3rd party content. 1st party content would be images, videos, or audio that is generated using a large online platform’s generative AI tools and is then posted or distributed on that platform. In this instance, a platform can actually control the creation of a content provenance or watermark into the content. As mentioned, many of our companies are already working to incorporate this type of technology to increase transparency around AI-generated content. It is currently technically infeasible to accurately and reliably detect content that is created using a different platform’s AI tools. As noted above, considering the current ease with which current watermarks can be broken, a legal requirement and mandate for 3rd party content isn’t appropriate.

Despite the identified limitations for such markings, the author argues the bill is the first step and is not intended to require perfection:

While no system is perfect, robust watermarking combined with behavioral analysis, red-teaming, and required disclosures will make deception far more difficult and increase the risk calculus of bad actors. The bill mandates ongoing improvements to standards as technology evolves to stay ahead of actors looking to circumvent these measures to cause harm.

Relevant here, Meta has recently committed to “label images that users post to Facebook, Instagram and Threads when we can detect industry standard indicators that they are AI-generated.”^[11] A group of tech companies, including Adobe, Google, and Microsoft, has established the Coalition for Content Provenance and Authenticity (C2PA) to address “the prevalence of misleading information online through the development of technical standards for certifying the source and history (or provenance) of media content.”^[12] OpenAI announced that it will add C2PA metadata to images created with ChatGPT and the API for the DALL-E 3 model.

Just last month, Google announced new advances in its watermarking technology, including the capability to watermark GenAI created text:

Among Google’s swath of new AI models and tools announced today, the company is also expanding its AI content watermarking and detection technology to work across two new mediums.

Google’s DeepMind CEO, Demis Hassabis, took the stage for the first time at the Google I/O developer conference on Tuesday to talk not only about the team’s new AI tools, like the Veo video generator, but also about the new upgraded SynthID watermark imprinting system. It can now mark video that was digitally generated as well as AI-generated text.

Watermarking AI-generated content will matter increasingly as the technology gains prevalence, especially when AI gets used for malicious purposes. It’s already been used to spread political misinformation, claim someone said something they haven’t, and create nonconsensual sexual content.

SynthID was announced last August and started as a tool to imprint AI imagery in a way that humans can’t visually decipher — but can be detected by the system. The approach is different from other aspiring watermarking protocol standards like C2PA, which adds cryptographic metadata to AI-generated content.

Google had also enabled SynthID to inject inaudible watermarks into AI-generated music that was made using DeepMind’s Lyria model. SynthID is just one of several AI safeguards in development to combat misuse by the tech, safeguards that the Biden administration is directing federal agencies to build guidelines around.^[13]

Many companies have already voluntarily committed to follow specified guidelines. As described in the White House fact sheet:

President Biden [convened] seven leading AI companies at the White House [] – Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI – to announce that the Biden-Harris Administration has secured voluntary commitments from these companies to help move toward safe, secure, and transparent development of AI technology.

Companies that are developing these emerging technologies have a responsibility to ensure their products are safe. To make the most of AI’s potential, the Biden-Harris Administration is encouraging this industry to uphold the highest standards to ensure that innovation doesn’t come at the expense of Americans’ rights and safety.

These commitments, which the companies have chosen to undertake immediately, underscore three principles that must be fundamental to the future of AI – safety, security, and trust – and mark a critical step toward developing responsible AI. As the pace of innovation continues to accelerate, the Biden-Harris Administration will continue to remind these companies of their responsibilities and take decisive action to keep Americans safe.^[14]

The most relevant commitment is focused on earning the public’s trust by ensuring individuals are aware of when content is AI generated:

Develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated, including robust provenance, watermarking, or both, for AI-generated audio or visual content

Companies making this commitment recognize that it is important for people to be able to understand when audio or visual content is AI-generated. To further this goal, they agree to develop robust mechanisms, including provenance and/or watermarking systems for audio or visual content created by any of their publicly available systems within scope introduced after the watermarking system is developed. They will also develop tools or APIs to determine if a particular piece of content was created with their system. Audiovisual content that is readily distinguishable from reality or that is designed to be readily recognizable as generated by a company’s AI system—such as the default voices of AI assistants—is outside the scope of this commitment. The watermark or provenance data should include an identifier of the service or model that created the content, but it need not include any identifying user information. More generally, companies making this commitment pledge to work with industry peers and standards-setting bodies as appropriate towards developing a technical framework to help users distinguish audio or visual content generated by users from audio or visual content generated by AI.^[15]

Similar legislation has been introduced at the federal level by Senators Schatz and Kennedy, the AI Labeling Act of 2023. Also at the federal level, President Biden issued an executive order late last year on AI, to set standards for safety and security and to ensure responsible AI development and deployment in the United States. The order defined generative AI as “the class of AI models that emulate the structure and characteristics of input data in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital content.”

The EO calls for the development of a risk management framework for generative AI, modeled after the AI Risk Management Framework created by the National Institute for Standards and Technology (NIST).

The many recent developments in this space are encouraging. However, given the expansive obligations imposed by the bill and the questions regarding feasibility for all of the GenAI providers, platforms, and manufacturers that must implement this technology, the author has agreed to amendments that delay the effective date of the various sections of the bill to July 1, 2026. This allows time for the technology to advance and also gives the Legislature a chance to make adjustments next year should they see fit.

1. Additional stakeholder positions

A coalition of groups in support, including SEIU California, writes:

Generative AI technologies are increasingly able to generate inauthentic images, audio, video, and text content—sometimes called “deepfakes”—that appear authentic even to experts. Fake content that is presented as real can include many things, including:

• Disinformation impacting free and fair elections, which we have already seen with deepfakes undermining or influencing national elections in Bangladesh, Slovakia, and Argentina, and local elections around the United States;
• Fake digital media that undermine national security, such as fake photos of an attack on the Pentagon, which were disseminated by a twitter account of RT, a Russian-backed media company, and which caused a US stock market plunge;
• Non-consensual intimate imagery, demonstrated by the obscene sexual imagery of Taylor Swift that was published and circulated widely around the time of the Super Bowl; and
• Many other forms of problematic content, including scams and fraud, child sexual abuse material, and plagiarism.

. . .

AB 3211, the California Provenance, Authenticity and Watermarking Standards (PAWS) Act, attempts to solve these problems. The bill will require AI content generators to provide the public with tools to distinguish between authentic and fake content across digital media. AI content generators will be required to embed maximally-indelible and privacy-protecting content provenance data into the content that they generate— whether AI-generated or authentic—and all large online platforms will be required to display clearly understandable labels on content that alert users to its provenance. While these requirements alone will not fully eliminate the harms caused by fake online content, they can dramatically reduce those harms by making clear which content is fake and which is real.

Oakland Privacy writes in opposition with concerns about the expansive scope of the bill, urging the author to consider focusing on particularly troublesome content rather than the broad blanket requirements imposed by the bill. They also advocate for more technology neutral language.

The Coalition in opposition also raises concerns about the enforcement mechanism:

Considering the prescriptive nature and technical infeasibility of some requirements and the technical impossibility of others, we believe the penalties for noncompliance of $1 million or 5% of global annual revenue are unjustifiable. As mentioned above, platforms cannot watermark text content. The technology to apply watermarks to audio, images, and videos are much further along but vary in their resistance to removal or inadvertent breaking.

The penalties imposed by the bill for any violation are extremely excessive. To more appropriately hone the penalty provision, the author has agreed to amendments that limit penalties to up to $50,000 per violation but provide for heightened penalties of up to $500,000 for violations that are intentional or that result from grossly negligent conduct.

SUPPORT

California Initiative for Technology & Democracy (sponsor)

Accountable Tech

Bay Rising

California Voter Foundation

Catalyst California

Center for Countering Digital Hate

Check My Ads

Chinese Progressive Association

City and County of San Francisco, Board of Supervisors

Hmong Innovating Politics

Move (Mobilize, Organize, Vote, Empower) the Valley

NextGen California

Partnership for the Advancement of New Americans

SEIU California

Techequity Action

OPPOSITION

California Chamber of Commerce

Computer and Communications Industry Association

Netchoice

Oakland Privacy

Technet

RELATED LEGISLATION

Pending Legislation:

SB 942 (Becker, 2024) places obligations on businesses that provide generative artificial intelligence (AI) systems to develop and make accessible tools to detect whether specified content was generated by those systems. These “covered providers” are required to include visible and imperceptible markings on AI-generated content to identify it as such. SB 942 is currently in the Assembly Privacy and Consumer Protection Committee.

SB 970 (Ashby, 2024) would have ensured that media manipulated or generated by AI technology is incorporated into the right of publicity law and criminal false impersonation statutes. SB 970 requires those providing access to such technology to provide a warning to consumers about liability for misuse. SB 970 was held in the Senate Appropriations Committee.

AB 1791 (Weber, 2024) requires social media platforms to delete “personal provenance data” related to a user's identity from content uploaded to the platform, while retaining provenance data related to the system or service used to generate the content. AB 1791 is currently in this Committee.

AB 2930 (Bauer-Kahan, 2024) requires, among other things, a deployer and a developer of an automated decision tool to, on or before January 1, 2026, and annually thereafter, perform an impact assessment for any automated decision tool the deployer uses that includes, among other things, a statement of the purpose of the automated decision tool and its intended benefits, uses, and deployment contexts. The assessments must be provided to the Civil Rights Department within 7 days of a request. AB 2930 requires a deployer to, at or before the time an automated decision tool is used to make a consequential decision, notify any natural person that is the subject of the consequential decision that an automated decision tool is being used to make, or be a controlling factor in making, the consequential decision and to provide that person with, among other things, a statement of the purpose of the automated decision tool. AB 2930 is currently in this Committee.

Prior Legislation:

SB 444 (Umberg, 2019) would have requested the Regents of the University of California (UC) to enact a resolution authorizing the law schools at UC Berkeley and UC Irvine to participate in a pilot project to develop AI or machine-learning solutions to address access to justice issues faced by self-representing litigants in their respective courts. The bill died in the Assembly Higher Education Committee. 

AB 1576 (Calderon, 2019) would have required the Secretary of GovOps to appoint participants to an AI working group to evaluate the uses, risks, benefits, and legal implications associated with the development and deployment of AI by California-based businesses. The bill was held on the Senate Appropriations Committee suspense file.

SJR 6 (Chang, Res. Ch. 112, Stats. 2019) urged the President and the Congress of the United States to develop a comprehensive AI Advisory Committee and to adopt a comprehensive AI policy.

PRIOR VOTES:

Assembly Floor (Ayes 62, Noes 0)

Assembly Appropriations Committee (Ayes 14, Noes 0)

Assembly Judiciary Committee (Ayes 10, Noes 0)

Assembly Privacy and Consumer Protection Committee (Ayes 10, Noes 0)

**************

1. Siddarth Srinivasan, Detecting AI fingerprints: A guide to watermarking and beyond (January 4, 2024) Brookings Institution, https://www.brookings.edu/articles/detecting-ai-fingerprints-a-guide-to-watermarking-and-beyond/#:~:text=Google%20also%20recently%20announced%20SynthID,model%20to%20detect%20the%20watermark. All internet citations are current as of June 12, 2024. ↑

2. Lonnie Lee Hood, Experts Say That Soon, Almost The Entire Internet Could Be Generated by AI (March 4, 2022) The Byte, https://futurism.com/the-byte/ai-internet-generation. ↑

3. Brian Contreras, Tougher AI Policies Could Protect Taylor Swift—And Everyone Else—From Deepfakes (February 8, 2024) Scientific American, https://www.scientificamerican.com/article/tougher-ai-policies-could-protect-taylor-swift-and-everyone-else-from-deepfakes/. ↑

4. Hannah Fry, Laguna Beach High School investigates ‘inappropriate’ AI-generated images of students (April 2, 2024) Los Angeles Times, https://www.latimes.com/california/story/2024-04-02/laguna-beach-high-school-investigating-creation-of-ai-generated-images-of-students. ↑

5. Em Steck & Andrew Kaczynski, Fake Joe Biden robocall urges New Hampshire voters not to vote in Tuesday’s Democratic primary (January 22, 2024) CNN, https://www.cnn.com/2024/01/22/politics/fake-joe-biden-robocall/index.html. ↑

6. Avalon Zoppo, Threat of AI-Generated ‘Deepfake’ Evidence Needs Judiciary’s Attention, Former Judge Says (October 27, 2023) The National Law Journal, https://www.law.com/nationallawjournal/2023/10/27/threat-of-ai-generated-deepfake-evidence-needs-judiciarys-attention-former-judge-says/?slreturn=20240303000917. ↑

7. Bobby Chesney & Danielle Citron, Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security (July 14, 2018) 107 California Law Review 1753 (2019), https://ssrn.com/abstract=3213954. ↑

8. See footnote 1. ↑

9. For privacy purposes, this provenance data shall not include personally identifiable information. ↑

10. The author has agreed to an amendment that makes reference to “social media platforms,” as that term is defined in existing law, rather than “social network.” ↑

11. Nick Clegg, Labeling AI-Generated Images on Facebook, Instagram and Threads (February 6, 2024) Meta, https://about.fb.com/news/2024/02/labeling-ai-generated-images-on-facebook-instagram-and-threads/. ↑

12. Overview, Coalition for Content Provenance and Authenticity, https://c2pa.org/. ↑

13. Umar Shakir, Google’s invisible AI watermark will help identify generative text and video (May 14, 2024) The Verge, https://www.theverge.com/2024/5/14/24155927/google-ai-synthid-watermark-text-video-io; see also Karissa Bell, Google expands digital watermarks to AI-made video and text (May 14, 2024) Yahoo!news, https://au.news.yahoo.com/google-expands-digital-watermarks-to-ai-made-video-175232320.html?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAAI9FrY1zdqIO5ttk0ewE40_6KKMExr2AatuMW87CU3d1m989vvDptiMDa208Ejy3PQ0NA1-2KuSNH1Po8zormuGzG2sldsFq4QBcl3vqj5EAw1TOmsAr25yuQwrak0rFONQQMNRT51cBTTamk8ox5fgtZPvQxfk9hnNEC2QnQETC. ↑

14. FACT SHEET: Biden-⁠Harris Administration Secures Voluntary Commitments from Leading Artificial Intelligence Companies to Manage the Risks Posed by AI (July 21, 2023) The White House, https://www.whitehouse.gov/briefing-room/statements-releases/2023/07/21/fact-sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial-intelligence-companies-to-manage-the-risks-posed-by-ai/. ↑

15. Ibid. ↑

ASSEMBLY THIRD READING

ATR Bill Id:AB 3211

Author:(Wicks)

As Amended Ver:April 18, 2024

Majority vote

SUMMARY

Establishes watermarking and content provenance requirements for generative artificial intelligence (GenAI) system providers, manufacturers of digital cameras and other recording devices, and large online platforms.

Major Provisions

1. Requires GenAI system providers to place imperceptible and maximally indelible watermarks containing provenance data into the content created by the provider's system; to provide downloadable watermark decoders that allow the public to quickly assess whether a given piece of content was created by the provider's system; to conduct red-teaming exercises involving third-party experts to test whether watermarks can be easily removed from the content produced by their systems, or falsely added to authentic content; to report any vulnerabilities or failures they discover in a system to the Department of Technology (CDT), any other system providers who may have been affected by similar vulnerabilities or failures, and any affected party whose contact information they have retained; and to produce Risk Assessment and Mitigation Reports that assess the risks posed by synthetic content and the harms that have been or could be caused by such content.
2. Prohibits any provider from making available a tool, application, or service designed to remove watermarks from synthetic content.
3. Prohibits GenAI hosting platforms from making available GenAI systems that do not place maximally indelible watermarks containing provenance data into content created by the systems.
4. Requires conversational AI systems to disclose to users that the system generates synthetic content.
5. Requires newly manufactured digital cameras and recording devices distributed in California to offer users the option to place watermarks into content produced by those devices.
6. Requires manufacturers of digital cameras to push software and firmware updates to existing digital cameras in order to grant them the ability to place watermarks on content, if technically feasible.
7. Requires large online platforms to label content posted to the platform with the provenance data embedded in the content; to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by AI systems without watermarking functionality; to require their users to disclose uploads of synthetic content; create a verification process for users to apply a digital signature to content created by human beings; and to produce Risk Assessment and Mitigation Reports that assess the risks posed by synthetic content.
8. Sets the penalty for a violation of the bill's provisions at the greater of $1,000,000 or 5% of the violator's annual global revenue, to be assessed as an administrative penalty by CDT. Requires CDT to adopt regulations as necessary within 90 days of the bill coming into effect.

COMMENTS

Background. In the past few years the rapid development of GenAI has allowed a vast quantity of convincing – but ultimately fake – content to be generated in an instant. When employed by bad actors, these capabilities have the potential to destroy lives and destabilize societies.

What this bill would do. This bill would require 1) the providers of GenAI systems to embed "watermarks" detailing content provenance into all GenAI ouputs; 2) digital cameras and other recording devices to offer users the option to embed watermarks into outputs; and 3) social media companies to prominently display labels that contain the provenance information of real and GenAI-produced content.

Analysis. As GenAI technologies become more accessible, online content that appears real, but that is actually false, threatens to flood social media and other large online platforms. The unmitigated spread of synthetic content threatens to harm individual Californians in numerous ways, such as through the proliferation of nonconsensual deepfake pornography, scams, and the distribution of targeted election disinformation. In principle, these issues can be tackled through a three-step plan: first, all artificial content must be labeled as such. Second, all "real" content – such as video and audio recordings – much be labeled as such. Third, large online platforms must be required to prominently display these labels, allowing users to distinguish between artificial and real content. This bill seeks to enact this plan.

According to the Author

The author states that their primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. According to the author, these include:

1. Harms caused by inauthentic content presented as authentic: The bill acknowledges the wide range of potential harms caused by inauthentic content, including financial scams, non-consensual intimate imagery, disinformation (especially around elections and public health), and the erosion of trust in the digital information ecosystem. The PAWS Act seeks to reduce these harms by requiring clear disclosure of content provenance, making it harder for inauthentic content to be mistaken as authentic.
2. Lack of transparency around provenance of digital media: The bill addresses transparency concerns by mandating that generative AI providers embed imperceptible and indelible watermarks containing provenance data in all synthetic content they create, and prominently display this provenance data to users. The bill also establishes standards for digital cameras and recording devices to offer watermarking options for authentic content.
3. Facilitation of harmful acts by bad actors: The bill prohibits the distribution of tools designed to remove watermarks or manipulate provenance data, making it more difficult for bad actors to generate unlabeled inauthentic content. It also requires conversational AI systems to disclose their artificial nature and obtain user consent, making it more difficult for bad actors to leverage AI generation for deception.

In sum, the bill seeks to establish clear standards and requirements around content provenance disclosure, watermarking, and labeling, with the goal of increasing transparency and reducing the ability of bad actors to deceive users with unlabeled synthetic content. By doing so, the author aims to mitigate the various potential harms enabled by increasingly sophisticated generative AI technologies.

Arguments in Support

The sponsor, CITED, writes:

California has the opportunity to lead the U.S. response to these threats. As the home of Silicon Valley and the heartland of AI innovation, California can and must play an important role in shaping the solution. To that end, AB 3211 represents a phased-in solution to address this complex problem that complements existing standard-setting efforts.

Arguments in Opposition

A coalition of industry associations including CalChamber writes:

AB 3211 seems to treat all AI-generated content as inherently bad or risky. By requiring such thorough and prescriptive requirements for content labeling, the bill makes a value judgment that consumers must be notified and aware of any content that was created by AI. This applies to all inauthentic content including purely artistic or satirical content. While we agree with the intent to provide more information to consumers, in some instances it could create disclosure or notification fatigue.

FISCAL COMMENTS

According to Assembly Appropriations Committee:

Costs to CDT (General Fund) likely in the low millions of dollars annually. CDT estimates costs of about $1.1 million in year one and $1.5 million in year two and ongoing for additional staff positions, external consultants, and costs of integrating GenAI within CDT.

VOTES

ASM PRIVACY AND CONSUMER PROTECTION: 10-0-1

YES: Bauer-Kahan, Joe Patterson, Bryan, Dixon, Irwin, Lowenthal, Ortega, Ward, Wicks, Wilson

ABS, ABST OR NV: Hoover

ASM JUDICIARY: 10-0-2

YES: Kalra, Dixon, Bauer-Kahan, Connolly, Haney, Maienschein, McKinnor, Pacheco, Reyes, Sanchez

ABS, ABST OR NV: Bryan, Waldron

ASM APPROPRIATIONS: 14-0-1

YES: Wicks, Sanchez, Arambula, Bryan, Calderon, Wendy Carrillo, Dixon, Mike Fong, Grayson, Haney, Hart, Jim Patterson, Pellerin, Villapudua

ABS, ABST OR NV: Ta

UPDATED

VERSION: April 18, 2024

CONSULTANT: Slater Sharp / P. & C.P. / (916) 319-2200 FN: 0003325

Date of Hearing: May 8, 2024

ASSEMBLY COMMITTEE ON APPROPRIATIONS

Buffy Wicks, Chair

ABPCA Bill Id:AB 3211

Author:(Wicks) – As Amended Ver:April 18, 2024

Urgency: No State Mandated Local Program: No Reimbursable: No

SUMMARY:

This bill implements watermark and content provenance requirements on artificial intelligence (AI) system providers, camera technology manufacturers, and online platforms, enforceable through administrative penalties assessed by the California Department of Technology (CDT).

Specifically, among other provisions, this bill:

1. Requires a generative AI provider to place imperceptible and maximally indelible watermarks into synthetic content produced or significantly modified by the provider’s generative AI system, develop and make available to the public watermark decoders that allow a user to determine whether a piece of content was created with the provider’s system, and conduct AI red-teaming exercises to test the efficacy of the provider’s watermarks.
2. Prohibits software providers and distributors from making available a system, application, tool, or service designed to remove watermarks from synthetic content.
3. Prohibits a generative AI hosting platform from making available a system that does not place maximally indelible watermarks into content created by the system.
4. Requires a conversational AI system to clearly and prominently disclose to users that the conversational AI system generates synthetic content.
5. Requires, commencing January 1, 2026, newly manufactured digital cameras and recording devices sold, offered for sale, or distributed in California to offer users the option to place an authenticity watermark and provenance watermark in the content produced by that device.
6. Requires, commencing March 1, 2025, a large online platform to use labels to prominently disclose the provenance data found in watermarks or digital signatures in content distributed to users on its platforms.
7. Requires a large online platform to require users to disclose whether uploaded content is synthetic content.
8. Authorizes CDT to impose administrative penalties of up to $1 million or 5% of a violator’s annual global revenue, whichever is greater, for violations of this bill.
9. Requires CDT to adopt regulations to implement this bill.

FISCAL EFFECT:

Costs to CDT (General Fund) likely in the low millions of dollars annually. CDT estimates costs of about $1.1 million in year one and $1.5 million in year two and ongoing for additional staff positions, external consultants, and costs of integrating GenAI within CDT.

COMMENTS:

1. Purpose. According to the author:

This bill seeks to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or “deep fake” content. Specifically, the bill aims to address the interrelated problems of artificial intelligence: 1. Harms caused by inauthentic content presented as authentic… 2. Lack of transparency around provenance of digital media… 3. Facilitation of harmful acts by bad actors.

1. Background. This bill establishes a framework for disclosure, watermarking, and labeling of digital content created or significantly modified by a generative or conversational AI system. Although images have long been manipulated or faked, the proliferation of AI systems that can generate and modify images and videos has dramatically increased the amount of convincing but ultimately fake content circulating in the public domain. This emerging technology has already resulted in creation of nonconsensual pornography, scams targeting individuals and corporations, and political propaganda and misinformation.
2. Support and Opposition. The sponsor of the bill, the California Initiative for Technology and Democracy, writes in support:

Ultimately, watermarking and provenance transparency are critical tools in our arsenal of defenses against the onslaught of AI-generated disinformation. AB 3211 is an urgent intervention that takes a dynamic approach to standard setting in an evolving technological context. For example, rather than codifying a static standard, the bill mandates that companies keep pace with the state of the art such that the quality of watermarks improves as the technology improves. Moreover, firms covered by the bill may comply with its requirements by conforming with emerging industry standards, such as the Content Credentials standard proposed by the Coalition for Content Provenance and Authenticity. Finally, compliance is made easier by the fact that large portions of AB 3211 mirror the upcoming requirements of the EU’s AI Act, which have been negotiated and agreed to after years of conversation with a variety of stakeholders, including the technology industry and its largest companies.

In opposition, TechNet writes:

[This bill] sets overly prescriptive and technological infeasible requirements on developers of artificial intelligence (AI), large online platforms, camera and recording device manufacturers to incorporate content provenance and watermarking technology into their products. We agree with the intent to create greater trust in user generated content online by fostering the adoption of content provenance verifications and watermarks. However, this bill presents a multitude of issues and requires platforms to comply with technically infeasible and impossible standards.

Analysis Prepared by: Annika Carlson / APPR. / (916) 319-2081

Date of Hearing: April 23, 2024

ASSEMBLY COMMITTEE ON JUDICIARY

Ash Kalra, Chair

ABPCA Bill Id:AB 3211

Author:(Wicks) – As Amended Ver:April 18, 2024

SUBJECT: California Provenance, Authenticity and Watermarking Standards

KEY ISSUES:

1. Should content produced utilizing artificial intelligence be required to utilize a watermark demonstrating its artificial origins and lack of veracity?
2. Should online platform companies that violate this bill be subject to administrative penalties of $1 million or 5 percent of their annual sales?

SYNOPSIS

Artificial intelligence technology presents a myriad of opportunities to better humanity. From predictive analytics in healthcare settings, to making workplaces more efficient, to making travel safer for all Americans, the benefits of artificial intelligence seem endless. However, there is a dark side to these technological advancements. Artificial intelligence can now produce lifelike fake images, video, and audio. Furthermore, predictive artificial intelligence technology can be utilized to scam unsuspecting consumers by learning their habits and then nefariously being used by criminals. While any person who has recently utilized a corporation’s artificial intelligence-based customer service portal may question the efficacy of this technology, the proponents of this measure highlight that increasingly sophisticated artificial intelligence is quickly proliferating in everyday life. Without significant safeguards, it will pose meaningful threats to society.

Seeking to combat the risks associated with artificial intelligence, this bill requires all content generated by artificial intelligence to be labeled as fake; camera manufacturers to offer the ability to imbed watermarks into images indicating their authenticity; and require online media platforms to label content posted on their platforms as authentic or fake. This measure would impose significant administrative penalties on entities that violate these requirements.

This measure is sponsored by the California Initiative for Technology & Democracy and is supported by labor organizations, legal aid organizations, and election reform advocates. The proponents of this measure highlight the growing threat of the unchecked deployment of artificial intelligence technology. They note that this bill will empower online media consumers to delineate between facts and falsehoods. The measure is strongly opposed by technology companies who contend that artificial intelligence is not inherently bad and that the requirements of this measure are unreasonably burdensome and impossible to comply with in a real world setting. This measure was previously heard and approved by the Committee on Privacy and Consumer Protection by a vote of 10-0.

SUMMARY: Regulates the use of generative artificial intelligence technology by requiring watermark and content provenance requirements on artificial intelligence system providers, camera technology manufacturers, and online platform companies. Specifically, this bill:

1. Requires a generative artificial intelligence provider to do all of the following:
   1. Place imperceptible and maximally indelible watermarks containing provenance data into synthetic content produced or significantly modified by a generative artificial intelligence system that the provider makes available;
   2. Develop downloadable watermark decoders that allow a user to determine whether a piece of content was created with the provider’s system, and make those tools available to the public; and
   3. Conduct artificial intelligence red-teaming exercises involving third-party experts to test whether watermarks can be easily removed from synthetic content produced by the provider’s generative artificial intelligence systems, as well as whether the provider’s generative artificial intelligence systems can be used to falsely add watermarks to otherwise authentic content.
2. Requires that if a sample of synthetic content is too small to contain the required provenance data as required by 1), the provider must, at minimum, attempt to embed watermarking information that identifies the content as synthetic and provide the following provenance information:
   1. The name of the generative artificial intelligence provider;
   2. The name and version number of the artificial intelligence system that generated the content;
   3. The time and date of the creation of the content; and
   4. If applicable, the specific portions of the content that are synthetic.
3. Requires a watermark decoder developed pursuant to 1) to comply with the following:
   1. Be easy to use by individuals seeking to quickly assess the provenance of a single piece of content; and
   2. Adhere, to the greatest extent possible, to relevant national or international standards.
4. Requires an entity conducting a red-teaming exercise pursuant to 1) to comply with the following:
   1. If a provider allows their generative artificial intelligence systems to be downloaded and modified, the provider must additionally conduct artificial intelligence red-teaming to assess whether their systems’ watermarking functionalities can be disabled;
   2. Make summaries of its red-teaming exercises publicly available in a location linked from the home page of the provider’s internet website, using a clearly labeled link that has a similar look, feel, and size relative to other links on the same web page; and
   3. Submit full reports of its red-teaming exercises to the Department of Technology within six months of conducting a red-teaming exercise.
5. Permits a generative artificial intelligence provider to continue to make available an artificial intelligence system that was made available before the date upon which this bill takes effect and that does not have watermarking capabilities, if either of the following conditions are met:
   1. The provider is able to retroactively create and make publicly available a decoder that accurately determines whether a given piece of content was produced by the provider’s system with at least 99 percent accuracy as measured by an independent auditor; or
   2. The provider conducts and publishes research to definitively demonstrate that the system is not capable of producing inauthentic content.
6. Prohibits providers and distributors of software from making available a system, application, tool, or service that is designed to remove watermarks from synthetic content.
7. Prohibits a generative artificial intelligence hosting platform from making available a system that does not place maximally indelible watermarks containing provenance data into content created by the system.
8. Requires a generative artificial intelligence hosting platform that discovers a vulnerability or failure in the system to report the issue to the Department of Technology within 24 hours of discover and do the following:
   1. Notify other generative artificial intelligence providers that may be affected by similar vulnerabilities or failures in a manner that allows the other providers to harden their own artificial intelligence systems against similar risks;
   2. Notify affected parties, including, but not limited to, online platforms, researchers or users who received incorrect results from a watermark decoder, or users who produced artificial intelligence content that contained incorrect or insufficient watermarking data; and
   3. Make any report to the Department of Technology publicly available in a location linked from the home page of the provider’s internet website with a clearly labeled link that has a similar look, feel, and size relative to other links on the same web page, as specified.
9. Requires a conversational artificial intelligence system to clearly and prominently disclose to users that the conversational artificial intelligence system generates synthetic content.
10. Requires, in visual interfaces, the disclosure specified in 9) to be placed in the interface itself and maintain the disclosure’s visibility in a prominent location throughout any interaction with the interface.
11. Requires, in audio-only interfaces, the disclosure specified in 9) to be verbally made at the beginning and end of a call.
12. Requires the disclosures made pursuant to 9) to be made in the language in which the communications are occurring.
13. Exempts from the provisions of 9) through 12) conversational artificial intelligence systems that do not produce inauthentic content.
14. Provides that the provisions of 1) through 13) take effect February 1, 2025.
15. Requires, commencing January 1, 2026, newly manufactured digital cameras and recording devices sold, offered for sale, or distributed in California to offer users the option to place an authenticity watermark and provenance watermark in the content produced by that device subject to the following requirements:
    1. A user has the option to remove the authenticity and provenance watermarks from the content produced by their device;
    2. Authenticity watermarks are to be turned on by default, while provenance watermarks are to be turned off by default;
    3. When a camera or audio recording application is open, a newly manufactured digital camera or recording device provides a clear indicator that a watermark is being applied;
    4. A newly manufactured digital camera or recording device allows the user to adjust the watermarks settings; and
    5. The manufacturer clearly informs a user of the existence of the authenticity and provenance watermarks settings upon the user’s first use of the camera or the recording function on the recording device, as specified.
16. Requires the watermarks required by 15) to comply with industry standards, as specified.
17. Requires, commencing January 1, 2026, a camera and recording device manufacturer to offer a software or firmware update enabling a user to place an authenticity watermark and provenance watermark on the content created by the device to a user of a digital camera or recording device purchased in California prior to January 1, 2026, if technically feasible.
18. Defines for the purposes of 15) though 17) the following:
    1. “Authenticity watermark” means a watermark of authentic content that includes the name of the device manufacturer;
    2. “Camera and recording device manufacturer” means the makers of a device that can record photographic, audio, or video content, including, but not limited to, video and still photography cameras, mobile phones with built-in cameras or microphones, and voice recorders; and
    3. “Provenance watermark” means a watermark of authentic content that includes details about the content, including, but not limited to, the time and date of production, the name of the user, details about the device, and a digital signature.
19. Requires, commencing March 1, 2025 a large online platform to use labels to prominently disclose the provenance data found in watermarks or digital signatures in content distributed to users on its platforms, as follows:
    1. The labels must indicate whether content is fully synthetic, partially synthetic, authentic, authentic with minor modifications, or does not contain a watermark;
    2. A user must be able to click or tap on a label to inspect provenance data in an easy-to-understand format; and
    3. The labels must be readily legible to an average viewer or, if the content is in audio format, must be clearly audible.
20. Requires a large online platform to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed or that was produced by generative artificial intelligence systems without watermarking functionality.
21. Requires a large online platform to require users to disclose whether uploaded content is synthetic content.
22. Requires a large online platform to include prominent warnings to users that uploading or distributing synthetic content without disclosing that it is synthetic content may result in disciplinary action.
23. Permits a large online platform to provide users with an option to indicate that the user is uncertain whether the content they are uploading or distributing is synthetic content, and that if a user uploads or distributes content and indicates that they are uncertain of whether the content is synthetic content, a large online platform must indicate that the uploaded or distributed content is possibly synthetic and display that indication in a manner that is visible or audible to viewers or listeners of the content.
24. Requires a large online platform to use state-of-the-art techniques to detect and label text-based inauthentic content that is uploaded by users.
25. Requires a large online platform to make accessible a verification process for users to apply a digital signature to authentic content, as specified.
26. Requires, commencing January 1, 2026, and annually thereafter generative artificial intelligence providers and large online platforms to produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated by their systems or hosted on their platforms.
27. Requires the Risk Assessment and Mitigation Report to include, but not be limited to, assessments of the distribution of artificially-generated child sexual abuse materials, nonconsensual intimate imagery, disinformation related to elections or public health, plagiarism, or other instances where synthetic or inauthentic content caused or may have the potential to cause harm.
28. Requires the Risk Assessment and Mitigation Report to be audited by qualified, independent auditors who assess and either validate or invalidate the claims made in the report.
29. Requires auditors reviewing a Risk Assessment and Mitigation Report to use state-of-the-art techniques to assess reports, and adhere to relevant national and international standards.
30. Authorizes administrative penalties of up to one million dollars ($1,000,000) or 5 percent of the violator’s annual global revenue, whichever is greater to be imposed by the Department of Technology for violations of this bill.
31. Requires, within 90 days of the bill taking effect the Department of Technology to adopt regulations to implement this bill.
32. Defines the following terms for the purpose of this bill:
    1. “AI red-teaming” means a structured testing effort to find flaws and vulnerabilities in an AI system, including, but not limited to, harmful or discriminatory outputs, unforeseen or undesirable system behaviors, limitations, or potential risks associated with misuse of the system;
    2. “Artificial intelligence” or “AI” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments;
    3. “Authentic content” means images, videos, audio, or text created by human beings without any modifications or with only minor modifications that do not lead to significant changes to the perceived contents or meaning of the content. Minor modifications include, but are not limited to, changes to brightness or contrast of images, removal of background noise in audio, and spelling or grammar corrections in text;
    4. “Conversational AI system” means chatbots and other audio- or video-based systems that can hold humanlike conversations through digital media, including, but not limited to, online calling, phone calling, video conferencing, messaging, application or web-based chat interfaces, or other conversational interfaces. Conversational AI systems include, but are not limited to, chatbots for customer service or entertainment purposes embedded in internet websites and applications;
    5. “Digital signature” means a digital method that allows a user to sign a piece of authentic or synthetic content with their name or device information, verifying that they created the content;
    6. “Generative AI hosting platform” means an online repository or other internet website that makes generative AI systems available for download;
    7. “Generative AI provider” means an organization or individual that creates, codes, substantially modifies, or otherwise produces a generative AI system;
    8. “Generative AI system” means an artificial intelligence system that generates derived synthetic content, including images, videos, audio, text, and other digital content;
    9. “Inauthentic content” means synthetic content that is so similar to authentic content that it could be mistaken as authentic;
    10. “Large online platform” means a public-facing internet website, web application, or digital application, including a social network, video sharing platform, messaging platform, advertising network, or search engine that had at least 1,000,000 California users during the preceding 12 months;
    11. “Maximally indelible watermark” means a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant industry standards;
    12. “Provenance data” means data that identifies the origins of synthetic content, as specified;
    13. “Synthetic content” means information, including images, videos, audio, and text, that has been produced or significantly modified by a generative AI system;
    14. “Watermark” means information that is embedded into a generative AI system’s output for the purpose of conveying its synthetic nature, identity, provenance, history of modifications, or history of conveyance; and
    15. “Watermark decoders” means freely available software tools or online services that can read or interpret watermarks and output the provenance data embedded in them.
33. Adopts a severability clause.

EXISTING LAW:

1. Defines “deepfake” to mean audio or visual content that has been generated or manipulated by artificial intelligence which would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do without their consent. (Government Code Section 11547.5.)
2. Requires social media companies to post terms of service for each social media platform owned or operated by the company in a manner reasonably designed to inform all users of the social media platform of the existence and contents of the terms of service. (Business and Professions Code Section 22676.)
3. Prohibits any person, firm, corporation or association, or any employee thereof with intent directly or indirectly to dispose of real or personal property or to perform services, professional or otherwise, or anything of any nature whatsoever or to induce the public to enter into any obligation relating thereto, to make or disseminate or cause to be made or disseminated before the public in this state, or to make or disseminate or cause to be made or disseminated from this state before the public in any state, in any newspaper or other publication, or any advertising device, or by public outcry or proclamation, or in any other manner or means whatever, including over the Internet, any statement, concerning that real or personal property or those services, professional or otherwise, or concerning any circumstance or matter of fact connected with the proposed performance or disposition thereof, which is untrue or misleading, and which is known, or which by the exercise of reasonable care should be known, to be untrue or misleading, or for any person, firm, or corporation to so make or disseminate or cause to be so made or disseminated any such statement as part of a plan or scheme with the intent not to sell that personal property or those services, professional or otherwise, so advertised at the price stated therein, or as so advertised. (Business and Professions Code Section 17500.)
4. Establishes the Department of Technology and tasks the Department with approval and oversight of information technology projects undertaken by the state. (Government Code Section 11545 et seq.)

FISCAL EFFECT: As currently in print this bill is keyed fiscal.

COMMENTS: As artificial intelligence technology becomes more widely available to average technology users, fears are growing that artificial intelligence can be utilized to manipulate existing images or audio or create realistic, yet fake, imagines, video, and audio. These manipulated images or audio can then be utilized to perpetrate scams, create phony content seeking to manipulate political discourse, or generate realistic looking pornographic content. Seeking to address these harms, this bill would require content generated by artificial intelligence to be labeled as “fake,” require camera makers to provide the ability for actual content to be deemed “real,” and require online platforms to display labels regarding the origins of content. In support of this measure, the author states:

This bill seeks to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or "deep fake" content. Specifically, the bill aims to address the interrelated problems of artificial intelligence:

1. Harms caused by inauthentic content presented as authentic: The bill acknowledges the wide range of potential harms caused by inauthentic content, including financial scams, non-consensual intimate imagery, disinformation (especially around elections and public health), and the erosion of trust in the digital information ecosystem. The PAWS Act seeks to reduce these harms by requiring clear disclosure of content provenance, making it harder for inauthentic content to be mistaken as authentic.

2. Lack of transparency around provenance of digital media: The bill addresses transparency concerns by mandating that generative AI providers embed imperceptible and indelible watermarks containing provenance data in all synthetic content they create., and prominently display this provenance data to users. The bill also establishes standards for digital cameras and recording devices to offer watermarking options for authentic content.

3. Facilitation of harmful acts by bad actors: The bill prohibits the distribution of tools designed to remove watermarks or manipulate provenance data, making it more difficult for bad actors to generate unlabeled inauthentic content. It also requires conversational AI systems to disclose their artificial nature and obtain user consent, making it more difficult for bad actors to leverage AI generation for deception.

In sum, the bill seeks to establish clear standards and requirements around content provenance disclosure, watermarking, and labeling, with the goal of increasing transparency and reducing the ability of bad actors to deceive users with unlabeled synthetic content. By doing so, the author aims to mitigate the various potential harms enabled by increasingly sophisticated generative AI technologies.

The proliferation of artificial intelligence technologies opens new avenues for bad actors to manipulate persons on the internet. In recent years, artificial intelligence technology has migrated from an expensive tool utilized primarily by corporations for high level computing processes and frustrating consumers in need of customer service assistance, to readily available products available to everyday consumers. As a result of the expansion of the use and availability of artificial intelligence, the risk associated with these tools has grown significantly. The author and proponents of this measure highlight three areas of pressing concern related to the democratization of artificial intelligence technologies: the technologies use in creating fake, yet lifelike pornography; the use of the technology in consumer scams; and the use of technology in undermining faith in democratic institutions.

In support of the need to better regulate artificial intelligence, the proponents of this measure highlight several recent incidents. Recently, three Southern California high schools saw students victimized by the dissemination of fake, but realistic looking pornographic videos depicting students. As high school students are known to do, these images were quickly spread around the student body creating significant mental distress for the nearly two-dozen students victimized by the fake videos. (Mackenzie Tatananni, “Inappropriate images” circulate at yet another California high school, as officials grapple with how to protect teens from AI porn created by classmates Daily Mail, (Apr. 11, 2024) available at: https://www.dailymail.co.uk/news/article-13295475/Inappropriate-images-California-Fairfax-High-School-AI-deepfake.html.)

Similarly, artificial intelligence (in concert with human actors) are creating a new type of online scams. For example, a scam known as “pig butchering” uses a combination of real people and artificial intelligence-powered chat bots to “befriend” social media users. These “friends” then try and convince the target of the scam to invest funds in fake online crypto currencies. When the victim then tries to cash in on the apparent gain from the crypto currency they discover they have been scammed by a chat bot that was able to use artificial intelligence to play to their emotions and steal their money. (Sam Hayson, John Oliver takes a deep dive into a murky online crypto scam, Mashable (Feb. 26, 2024) available at: https://mashable.com/video/john-oliver-pig-butchering-online-scam.)

The final, and potentially most concerning use of artificial intelligence, involves the creation of fake content designed to influence elections. Fake content can be utilized to make it appear that a candidate said something offensive, adopted a policy position contrary to reality, or look unfit for office. Furthermore, this threat is not hypothetical. During the recent New Hampshire Presidential Primary, one study suggested that between 5,000 and 20,000 New Hampshire residents received artificially-generated phone calls impersonating President Biden that told them not to vote in the state’s primary. (Adam Edelman, States turn their attention to regulating AI and deepfakes as 2024 kicks off, NBC News, (Jan. 22, 2024) available at: www.nbcnews.com/politics/states-turn-attention-regulating-ai-deepfakes-2024-rcna135122.) As the United States faces an incredibly contentious rematch between President Joe Biden and Donald Trump, one can only imagine that the threat of fake online content designed to influence the election will only grow.

This bill seeks to provide online media consumers with tools to assist in differentiating between real and fake content. In essence, this bill seek to combat artificial intelligence in three ways. First, this measure requires all content generated by artificial intelligence to be labeled as fake. Secondly, this measure requires manufacturers of camera technology to ensure that the content producer by their devices can be labeled as authentic. Finally, this measure requires large online platforms, defined as meaning a public-facing internet website, web application, or digital application, including a social network, video sharing platform, messaging platform, advertising network, or search engine that had at least 1,000,000 California users during the preceding 12 months, to properly apply these labels to content posted on their platforms. Violations of this measure would subject entities to administrative penalties of $1 million or five percent of a violator’s annual global revenue per violation.

It should be noted that this bill was previously analyzed by the Committee on Privacy and Consumer Protection. As the Committee with primary jurisdiction over technology issues, that Committee’s analysis provides a thorough review of the technology related to artificial intelligence and the watermarking required to produce the labels required by this bill. Accordingly, the remainder of this analysis will focus on the legal issues posed by this measure. Parties interested in a more comprehensive review of the underlying technical aspects of this measure are strongly encouraged to review the Committee on Privacy and Consumer Protection’s analysis of this measure which is available here: https://leginfo.legislature.ca.gov/faces/billAnalysisClient.xhtml?bill_id=202320240AB3211#.

By holding online platforms accountable for third party content, this bill may face federal preemption concerns. The federal Communications Decency Act provides that “[n]o provider or user of an interactive computer service shall be treated as the publisher or speaker of any information provided by another information content provider,” and affords broad protection from civil liability for the good faith content moderation decisions of interactive computer services. (47 U.S.C. Sec. 230(c)(1) and (2).) Though Section 230 was originally passed in response to judicial inconsistency with respect to the liability of internet service providers under statutes pertaining to “publishers” of content created by others, it has since been interpreted to confer operators of social media platforms and other online services with broad immunity from liability for content posted on their platforms by others.

Section 230 also indicates that “[n]othing in this section shall be construed to prevent any State from enforcing any State law that is consistent with this section,” but further provides that “[n]o cause of action may be brought and no liability may be imposed under any State or local law that is inconsistent with this section.” (47 U.S.C. Sec. 230(e)(3).) The latter provision has generally been interpreted to expressly preempt any state law that has the effect of treating a social media or other online platform as the publisher of information posted by other users, including prescriptive requirements relating to content moderation. This is consistent with the law’s original intent, which was to ensure that internet platforms facilitating the sharing of content can do so without considerable risk of liability in the event that content is not meticulously policed.

Since the development of social media platforms, the extent of Section 230’s immunity shield has been heavily litigated. Questions have arisen regarding what elements of a user’s experience and therefore of a social media platform’s business model, benefit from Section 230, and which fall outside its scope. In Fair Housing Council v. Roommates.com, LLC, Roommates.com was sued by a coalition of fair housing councils in California for allegedly violating the federal Fair Housing Act and state housing discrimination laws. (Fair Housing Council of San Fernando Valley v. Roommates.com, LLC (2007) 521 F.3d 1157.) The claim was based largely on the website’s search function which appeared to prefer certain profiles over others, seemingly on the basis of elements of the user’s identity, including their gender and sexual orientation, which were collected by Roommate.com through a mandatory questionnaire developed by the website. The District Court ruled in favor of Roommates.com, holding that the website was protected by Section 230. The Fair Housing Council subsequently appealed to the Ninth Circuit, which held that the website’s use of the information they required users to submit in order to utilize the website rendered Roommate.com outside the protection of Section 230. In its analysis, the court reasoned that a “website operator can be both a service provider and a content provider: If it passively displays content that is created entirely by third parties, then it is only a service provider with respect to that content. But as to content that it creates itself, or is ‘responsible, in whole or in part’ for creating or developing, the website is also a content provider.” (Id. at p. 1163.) With regard to Roommate.com’s own role in developing the users’ profiles, the court argued, “Roommate is ‘responsible’ at least ‘in part’ for each subscriber’s profile page, because ever such page is a collaborative effort between Roommate and the subscriber.” (Id. at p. 1167.)

Most recently, the Supreme Court issued a decision in a pair of cases addressing the same practice – Twitter, Inc. v. Taamneh and Gonzalez v. Google. In that decision, the Supreme Court held that “the failure to allege that the platforms here do more than transmit information by billions of people – most of whom use the platforms for interactions that once took place via mail, on the phone, or in public areas – is insufficient to state a claim that defendants knowingly gave substantial assistance and thereby aided and abetted ISIS’ acts. A contrary conclusion would effectively hold any sort of communications provider liable for any sort of wrongdoing merely for knowing that the wrongdoers were using its services and failing to stop them.” (Twitter, Inc. v. Taamneh, (2023) 143 S. Ct. 1206, 1213.)

To the extent that the decision in Twitter left room for argument that social media platforms may still be held liable for the distribution of CSAM due to the exemption found at (e)(1) of the code, it is likely that a recent denial of a petition for certiorari foreclosed that argument. Section 230 includes an exemption to immunity for enforcement of sex trafficking laws, including the sexual exploitation of children. (47 U.S.C. Sect. 230(1)(e).) In Does v. Reddit, (2022) 51 F.4^th 1137, a group of parents brought a claim against Reddit after discovering explicit images or videos of their children posted to various subreddit groups. The parents claimed that, despite contacting the platform with the information countless times, Reddit did little if anything to remove the content or prevent it from being shared due to the profit it provides the platform. (Reddit, 51 F.4th 1137, 1139.) Based on this allegation, the parents argued that Reddit reaped financial benefit from openly hosting child pornography and sought recovery under federal law 18 U.S.C. Section 1495, claiming that Reddit is liable as a beneficiary of child sex trafficking, among other causes of action. (Id. at p. 1140.) The district court held that in order to trigger the exemption to immunity provided by Section 230, plaintiffs seeking to hold a platform liable must plead that Reddit’s own conduct violated the criminal sex trafficking statute, a requirement which the plaintiffs failed to meet. (Ibid.)

Applying these cases to this bill, several distinctions are notable. First, the mandate for labeling content ultimately falls onto the online platform itself. To the extent that content posters must upload content with the various watermarks that is because the bill compels online platforms to demand such content. To the extent that an affirmative duty to label content is being placed on the online platform, this bill is not necessarily holding the platform liable for the underlying content itself. In that sense the bill may clear Section 230 preemption. Nonetheless, as discussed above, federal courts are broadly interpreting Section 230. While this bill is laudable and creatively seeking to address a pressing national issue, it may nonetheless run afoul of Section 230, especially in light of the Supreme Court’s broad application of that statute.

The sizable administrative penalties proposed by this measure may implicate the Eighth Amendment’s prohibition on unreasonable fines. Notwithstanding any consideration of Section 230 of the Communications Decency Act, the sizable administrative penalties imposed by this bill warrant a review of the Eighth Amendment’s prohibition on unreasonable fines. The prohibition against unreasonable fines has been applied to government imposed penalties for over two decades. (Hudson v. United States (1997) 522 U.S. 93.) Given that this measure authorizes penalties up to $1 million or five percent of a violator’s annual global revenue, this bill certainly imposes significant penalties. To determine if these sizable penalties are reasonable, or unlawfully excessive, the case law in this area generally requires a court to look at the proportionality of the penalty amount in comparison to the underlying violation. (United States v. Bajakajian (1998) 524 U.S. 321.)

As discussed above, the potential harms stemming from a violation of this measure are significant. Ranging from proliferating life-like, yet fake, child pornography, to defrauding persons out of tens of thousands of dollars, to threatening American democracy, the implications of failing to reign in artificially created content are significant. Additionally, given that the case precedent surrounding the Eighth Amendment’s prohibition on excessive fines typically examines fines on a case-by-case basis, drawing a broad conclusion about the constitutionality of this bill is difficult. Given the harms this bill seeks to combat, a $1 million per violation fine seems reasonable, especially if those harms come to pass. However, the annual revenue piece penalty aspect of this bill may be excessive, especially for violations that do not relate to the immediate undermining of American democracy (in those instances this penalty may be more than reasonable). Accordingly, should this bill advance, in order to lessen the risk of a constitutional challenge the author may wish to consider amendments to limit the fines imposed by this bill to the $1 million per violation only. One would hope that the threat of a $1 million per violation fine would be sufficient to compel online platforms to comply with this measure.

By requiring parties to add various labels to online content this bill represents government compelled speech. In addition to the Eighth Amendment, by requiring the inclusion of various watermarks and disclosures in online content, this bill also implicates the First Amendment and government compelled speech provisions. Because the speech contemplated by this bill is generally being compelled upon commercial entities, a business speech analysis appears to be most appropriate for this bill. When the government is compelling commercial entities to engage in speech, a review of such obligations is subject to an intermediate scrutiny level of review. (Virginia State Board of Pharmacy v. Virginia Citizens Consumer Council, Inc. (1976) 425 U.S. 748.) As it specifically relates to commercial speech, the United States Supreme Court has adopted a three-part test to determine the legitimacy of the government compelling a private company to place notices on products. This test, commonly referred to as the Central Hudson test, requires the government to demonstrate the following to be able to justify compelling speech: 1) the government has a substantial interest in the matter; 2) the regulation must directly and materially advance the government’s substantial interest; and 3) the government’s actions are narrowly tailored to advance that interest. (Central Hudson Gas and Electric Corp. v. Public Service Commission (1980) 447 U.S. 557.)

Applying the Central Hudson test to this measure, the bill appears to pass constitutional review. First, the government has a substantial interest in protecting Americans from the proliferation of false content on the internet. When looking at the concerns raised by the proponents, the government certainly has an interest in combatting fake child porn, stopping harmful scams, and protecting the integrity of our electoral systems. As to the second prong of the test, this bill appears to be crafted in such a way as to advance the government’s interest. Requiring disclosure of the source of online content would permit the public to assess the veracity of images and statements on the internet. Finally, as to this bill being narrowly tailored, again, the bill appears to meet the constitutional standard. Requiring the application of a watermark, while compelled speech, is a relatively simple and easy task to achieve compliance with this measure. The watermark appears easy to create and would give the public a clear signal as to the veracity of online content. One could easily envision, and this Legislature has previously proposed, far more burdensome requirements on online platforms seeking to combat misinformation. Accordingly, this bill does not appear to unlawfully compel speech.

In light of American’s longstanding propensity to believe and propagate conspiracy theories, a watermark may not be enough to combat deep fakes and other misleading online content. A troubling trend in recent years is American’s willingness to believe almost anything they see or read on the internet. Recognizing this tendency, conspiracy theorists, particularly those with underlying political aims, have poisoned the thoughts of millions of Americans to make them discount events they witness with their own eyes. While this bill seeks to combat these falsehoods, the ability of a watermark to convince a person that an image is false (especially when that person is inclined to believe its authenticity) may be insufficient.

For example, millions of Americans watched the events of January 6, 2021 unfold on live television. To anybody who watched those events, there should be no question that a group of insurrectionists stormed the United States Capitol seeking to prevent the rightful election of the candidate they did not prefer and killed or injured dozens of law enforcement officers in the process. Nonetheless, thanks in large part to online misinformation and the constant lies told by some television personalities, 25 percent of Americans now dispute the events of that day. (Jackman et al, A quarter of Americans believe FBI instigated Jan. 6, Post-UMD poll finds, The Washington Post (Jan. 4, 2024) available at: https://www.washingtonpost.com/dc-md-va/2024/01/04/fbi-conspiracy-jan-6-attack-misinformation/.) A similar number of conspiracy-minded Americans dispute the events of September 11, 2001, arguably one of the worst days in American history. Despite the fact that most of the nation watched the events of that horrible day in real time, a 2019 survey found that 23 percent of Americans either “strongly” or “somewhat” believe that the September 11^th attacks were an “inside job.” (https://www.statista.com/statistics/959504/belief-september-11-inside-job-conspiracy-us/.)

To Americans who distrust authority it is highly unlikely that a watermark will dissuade them from the accuracy of a statement they are inclined to believe, and more troubling, dissuade that person from further sharing online disinformation. Nonetheless, to the extent this bill helps stop misinformation from reaching the depths of the internet in the first places, this bill is laudatory.

ARGUMENTS IN SUPPORT: This bill is sponsored by the California Initiative for Technology & Democracy and is supported by labor organizations, legal aid organizations, and election reform advocates. In support of this bill the California Initiative for Technology & Democracy states:

Generative artificial intelligence (GenAI) technologies are powerful tools that are capable of creating all manners of images, audio, video, and text content from simple prompts. The breakneck speed at which these tools have evolved has meant human beings are increasingly unable to tell the difference between authentic, human-generated content, and synthetic content generated by AI.

The impact of the increasingly blurry line between authentic and synthetic digital media is already being felt by our society. From supercharging online scams, to using child sexual abuse material to generate non-consensual intimate imagery, to the proliferation of public safety4 and political disinformation, GenAI tools have contributed to the steady erosion of trust in our information ecosystem. Without adequate tools to help differentiate between human-generated content and AI-generated content, the truth decay already happening in our society will only accelerate. The detrimental impact of these AI tools represents a monumental challenge for our society that will require multi-faceted solutions that recognize and respond to the dynamic nature of evolving technology.

AB 3211 represents a phased-in solution to address this complex problem that complements existing standard-setting efforts. The bill has been developed with insights from EU regulators who have worked intimately on the AI Act and from federal policy experts who see, in the face of congressional inaction, California’s power to drive nationwide change. This bill makes critical interventions at two stages of our information system, first at the point of content creation and then at the point of dissemination. At the point of content creation, AB 3211 would enable human-created authentic content to be differentiated from AI-generated synthetic content by requiring separate watermarking standards for each. AB 3211 would require cameras and recording devices sold in California to include an option to place an authenticity watermark on the content that the device produces.

Separately, AI companies would be required to place imperceptible and maximally-indelible watermarks on any content that their AI systems produce. These watermarks would contain provenance information, allowing users, social media platforms, and the public to readily trace the source of the content. AB 3211 would also impose rigorous testing and reporting standards, as well as impact assessments, to ensure watermarks are resistant to manipulation and removal. Ultimately, watermarking and provenance transparency are critical tools in our arsenal of defenses against the onslaught of AI-generated disinformation. AB 3211 is an urgent intervention that takes a dynamic approach to standard setting in an evolving technological context.

ARGUMENTS IN OPPOSITION: This bill is opposed by several advocates for technology companies, including TechNet and the Computer & Communications Industry Association, as well as the California Chamber of Commerce. An opposition letter submitted by the entire coalition opposing this measure states:

We agree with the intent to create greater trust in user generated content online by fostering the adoption of content provenance verifications and watermarks. However, this bill presents a multitude of issues and requires platforms to comply with technically infeasible and impossible standards.

Many of our companies and platforms are at the forefront of developing content provenance and watermarking technology, which is still in its early stages. However, AB 3211 enacts incredibly prescriptive requirements for a technology that is still under development and rapidly evolving. For example, there isn’t a program that can watermark text, making the bill’s requirements to do so impossible to comply with. We believe references to text watermarking should be removed to reflect this reality.

Furthermore, content provenance and watermarking is still incredibly unreliable and in many cases easy to break. Researchers at the University of Maryland were able to break all the currently available watermarking methods. Some can be avoided by simple cropping, resizing, or screenshotting an image. More concerning, these researchers were able to insert fake watermarks and credentials into images, creating false positives. Provenance and watermarking tools tend to help good faith actors act virtuously, but they have limits on stopping bad actors. No provenance solution that’s been created so far, including watermarking or metadata, stops bad actors from simply ‘stripping’ provenance elements and posting a fake piece of content as authentic.

In its standards for large online platforms, AB 3211 should more clearly delineate between 1st party and 3rd party content. 1st party content would be images, videos, or audio that is generated using a large online platform’s generative AI tools and is then posted or distributed on that platform. In this instance, a platform can actually control the creation of a content provenance or watermark into the content. As mentioned, many of our companies are already working to incorporate this type of technology to increase transparency around AI-generated content. It is currently technically infeasible to accurately and reliably detect content that is created using a different platform’s AI tools. As noted above, considering the current ease with which current watermarks can be broken, a legal requirement and mandate for 3rd party content isn’t appropriate.

Moreover, AB 3211 requires platforms to use “state-of-the-art techniques” to detect and label inauthentic content that is uploaded by users. As a threshold matter, “state-of-the-art techniques” is a term of art used several times throughout the bill and has no legal meaning. It should be struck in favor of a clear standard, especially considering the exorbitant penalties the bill would seek to levy for any violation ($1 million or 5% of global annual revenue, whichever is higher). Section (e)(2) authorizes platforms to use privacy intrusive methods to detect inauthentic text content including spyware and user authentication. The bill allows platforms to consider a user’s typing cadence, which is only possible with spyware that can measure and record that information. The bill also allows a platform to verify that users are matched to their unique device identifier such as a subscriber identity module (SIM) card or multifactor authentication (MFA). This seems to give platforms the authority to destroy users’ anonymity in the pursuit of detecting and labeling inauthentic content. To be clear, our platforms are firmly against such invasive measures into our users’ information. It is a highly concerning precedent that California would be setting.

AB 3211 seems to treat all AI-generated content as inherently bad or risky. By requiring such thorough and prescriptive requirements for content labeling, the bill makes a value judgment that consumers must be notified and aware of any content that was created by AI. This applies to all inauthentic content including purely artistic or satirical content. While we agree with the intent to provide more information to consumers, in some instances it could create disclosure or notification fatigue. If watermarks and content credentials become so routine and placed on all AI-generated content, users may start to ignore and disregard their presence. Rather than focusing on whether the content itself was AI-generated, synthetic, or inauthentic, we would advise focusing on the misuse of this technology.

REGISTERED SUPPORT / OPPOSITION:

Support

Asian Americans and Pacific Islanders for Civic Empowerment
Asian Law Alliance
Bay Rising
California Clean Money Campaign
California Initiative for Technology & Democracy, a Project of California Common CAUSE
California Voter Foundation
Chinese Progressive Association
Courage California
Hmong Innovating Politics
Partnership for The Advancement of New Americans
SEIU California

Opposition

California Chamber of Commerce
Computer & Communications Industry Association
Netchoice
Technet

Analysis Prepared by: Nicholas Liedtke & Manuela Boucher- de la Cadena / JUD. / (916) 319-2334

Date of Hearing: April 16, 2024

ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION

Rebecca Bauer-Kahan, Chair

ABPCA Bill Id:AB 3211 (

Author:Wicks) – As Amended Ver:March 21, 2024

AS PROPOSED TO BE AMENDED

SUBJECT: California Provenance, Authenticity and Watermarking Standards

SYNOPSIS

As generative AI (GenAI) technologies become more accessible, online content that appears real, but that is actually false, threatens to flood social media and other large online platforms. The unmitigated spread of synthetic content threatens to harm individual Californians in numerous ways, such as through the proliferation of nonconsensual deepfake pornography, scams, and the distribution of targeted election disinformation.

The issue of GenAI-produced content can be tacked through a three-step plan. First, all artificial content must be labeled as such. Second, all “real” content – such as video and audio recordings – much be labeled as such. Third, large online platforms must be required to prominently display these labels, allowing users to distinguish between artificial and real content.

This bill seeks to enact the plan described above. It would require 1) the providers of GenAI systems to embed “watermarks” detailing content provenance into all GenAI ouputs; 2) digital cameras and other recording devices to offer users the option to embed watermarks into outputs; and 3) social media companies to prominently display labels that contain the provenance information of real and GenAI-produced content.

This bill is sponsored by CITED and supported by Asian Americans and Pacific Islanders for Civic Empowerment, the Asian Law Alliance, the California Voter Foundation, and Courage California. It is opposed by the California Chamber of Commerce, Computer & Communications Industry Association, Netchoice, and Technet.

Proposed committee amendments revise and recast the bill to make certain substantive changes, detailed in the analysis, as well as other technical and clarifying changes. If the bill passes this Committee, it will next be heard by the Judiciary Committee.

SUMMARY: Establishes watermarking and content provenance requirements for GenAI system providers, manufacturers of digital cameras and other recording devices, and large online platforms. Specifically, this bill:

1. Requires GenAI system providers to place imperceptible and maximally indelible watermarks containing provenance data into the content created by the provider’s system.
2. Defines “generative AI system” to mean the class of AI models that emulates the structure and characteristics of input data to generate derived synthetic content, including images, videos, audio, text, and other digital content.
3. Defines “generative AI system providers” to mean organizations or individuals that make AI systems, or substantial components thereof, available on the market, put them into service, provide them as standalone models or embed them in other systems or products, or provide them under free and open source licenses as a service, or through other large online platforms. GenAI system distributors include repositories or hosting internet websites that make AI systems available for download, even if those repositories are not the original makers of the AI systems they make available, and providers of conversational AI systems.
4. Defines “watermark” to mean information that is embedded into a generative AI output for the purposes of verifying the authenticity of the output or the identity or characteristics of its provenance, modifications, or conveyance.
5. Defines “provenance data” to mean data that includes the name of the AI system that generated the content, the underlying AI models that were part of the AI system, the time and date of the creation of the content, and, if applicable, which specific portions of the content are synthetic.
6. Requires providers to provide downloadable watermark decoders that allow individuals or large online platforms to quickly assess whether a given piece of content was created with the provider’s system, and to make those tools available to all large online platforms and the public. Defines “watermark decoders” to mean freely available software tools or online services that can read or interpret watermarks and output the provenance data embedded in them.
7. Requires providers to conduct red-teaming exercises involving third-party experts to test whether watermarks can be easily removed from the content produced by their systems, or falsely added to authentic content.
8. Requires summaries of these exercises to be posted online and sent to the Department of Technology (CDT) within six months of the bill taking effect, and annually thereafter.
9. Defines “AI red-teaming” to mean a structured testing effort to find flaws and vulnerabilities in an AI system, including, but not limited to, harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associated with misuse of the system.
10. Defines “authentic content” to mean images, videos, audio clips, or text created by human beings without any modifications or with only minor modifications that do not lead to significant changes to the perceived contents or meaning of the content.
11. Prohibits providers from continuing to support systems that were in use prior to this bill taking effect unless either of the following conditions is met:
12. The provider releases a watermark decoder tool that can determine whether a given piece of content was created by the provider’s GenAI system.
13. The provider publishes research demonstrating that the content produced by their system could not reasonably be mistaken as authentic.
14. Prohibits any provider or distributor of software or online services from making available or distributing a tool, application, or service capable of removing watermarks from synthetic content. Defines “synthetic content” to mean information, including images, videos, audio clips, and text, that has been significantly modified or generated by algorithms, including by AI.
15. Prohibits the distribution of open-source GenAI systems that are capable of having their watermarking functions removed.
16. Requires providers to report any vulnerabilities or failures they discover in a system to CDT, any other system providers who may have been affected by similar vulnerabilities or failures, and any affected party.
17. Requires providers to post any report made to the CDT online. If the publication of the report could pose public safety risk, allows the provider to instead post a summary disclosure or delay publication of the full report for no longer than 30 days.
18. Requires conversational AI systems to disclose to users that they receive synthetic content. For visual interfaces, this disclosure is required to be displayed prominently throughout a user interaction. For audio interfaces, the disclosure is required to be made verbally. Requires conversational AI systems to obtain affirmative consent from users before beginning a conversation. Defines “conversational AI systems” to mean chatbots and other audio- or video-based systems that can hold humanlike conversations through digital media, including, but not limited to, online calling, phone calling, video conferencing, messaging, application or web-based chat interfaces, or other conversational interfaces.
19. Beginning January 1, 2026, requires newly manufactured digital cameras and recording devices distributed in California to offer users the option to place authenticity and provenance watermarks in content produced by those devices. Requires that those watermarks abide by relevant industry standards.
20. Defines “authenticity watermark” to mean a watermark of authentic content that includes the name of the device manufacturer.
21. Defines “provenance watermark” to mean a watermark of authentic content that includes details about the content, including, but not limited to, the time and date of production, the name of the user, details about the device, and a digital signature.
22. Requires authenticity and provenance watermarks to persist in content produced using third-party applications that grant access to a digital camera’s functionality.
23. Requires manufacturers of digital cameras to push software and firmware updates to existing digital cameras in order to grant them the ability to place watermarks on content, if technically feasible.
24. Beginning March 1, 2025, requires a large online platform to label content posted to the platform with the provenance data embedded in the content. The labels must indicate whether the content is fully synthetic, partially synthetic, authentic, authentic with minor modifications, or does not contain a watermark. Defines “large online platform” to mean a public-facing internet website, web application, or digital application, including a social network, video sharing platform, messaging platform, advertising network, or search engine that had at least 1,000,000 California users during the preceding 12 months.
25. Requires large online platforms to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed and that was produced by AI systems without watermarking functionality.
26. Requires large online platforms to require their users to disclose uploads of synthetic content. Requires these platforms to warn users that failing to disclose the nature of such content will lead to disciplinary action.
27. Permits a platform to give users the option to indicate they are unsure of whether uploaded content is synthetic. If so indicated, requires platforms to label the content “possibly synthetic”.
28. Requires platforms to disable the accounts of users that have uploaded or distributed inauthentic content without disclosing it.
29. Requires platforms to create a verification process for users to apply a digital signature to content created by human beings. Requires the process to include options to verify without requiring disclosure of personally identifiable information.
30. Requires GenAI providers, GenAI distributors, and large online platforms to produce Risk Assessment and Mitigation Reports that assess the risks posed by synthetic content and the harms that have been or could be caused by such content.
31. Requires, at a minimum, that these reports include assessments on the distribution of AI-generated child sexual abuse materials, nonconsensual pornography, election and public health disinformation, and plagiarism.
32. Requires that these reports be audited by independent auditors using state-of-the-art techniques.
33. Defines GenAI system distributors to mean organizations or individuals that distribute GenAI systems, or substantial components thereof, such as model weights, in ways that can be downloaded and used by individuals locally on their own hardware, or modified or incorporated into other products or services.
34. Permits CDT to authorize independent researchers to access special researcher tools designed to facilitate large-scale and efficient analysis of AI-generated content.
35. Sets the penalty for a violation of the bill’s provisions at the greater of $1,000,000 or 5% of the violator’s annual global revenue, to be assessed as an administrative penalty by CDT.
36. Requires CDT to adopt regulations as necessary within 90 days of the bill coming into effect. Requires CDT to adopt specific national or international standards for provenance, authenticity, watermarking, or digital signatures.
37. Provides that the bill’s provisions are severable.

EXISTING LAW:

1. Establishes CDT and tasks it with the approval and oversight of information technology projects undertaken by the state. (Gov. Code § 11545 et seq.)
2. Defines “deepfake” to mean audio or visual content that has been generated or manipulated by artificial intelligence (AI) which would falsely appear to be authentic or truthful and which features depictions of people appearing to say or do things they did not say or do without their consent. Requires the Secretary of Government Operations to evaluate the impact of the proliferation of deepfakes on the state. (Gov. Code § 11547.5.)
3. Requires social media companies to post terms of service for each social media platform owned or operated by the company in a manner reasonably designed to inform all users of the social media platform of the existence and contents of the terms of service. (Bus. & Prof. § Code 22676.)

FISCAL EFFECT: As currently in print, this bill is keyed fiscal.

COMMENTS:

1) AI and GenAI. The development of GenAI is creating exciting opportunities to grow California’s economy and improve the lives of its residents. GenAI can generate compelling text, images and audio in an instant – but with novel technologies come novel safety concerns.

What is AI? In brief, AI is the mimicking of human intelligence by artificial systems such as computers. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. AI is not fundamentally different from other computer functions; its novelty lies in its application. Unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.

AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from GenAI, which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users. When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.

GenAI tools can be released in open-source or closed-source formats by their creators. Open-source tools are publically available; researchers and developers can access their code and parameters. This accessibility increases transparency, but it has downsides: when a tool’s code and parameters can be easily accessed, they can be easily altered, and open-source tools have the potential to be used for nefarious purposes such as generating deepfake pornography and targeted propaganda. Some open source models currently on the market include:

• LLaMA2: “Large Language Model Meta AI 2”. An open source LLM released by Meta in 2023.
• Bert: “Bidirectional Encoder Representations from Transformers”. An open source LLM released by Google in 2018.
• Stable Diffusion: An open source image generator released by startup Stability AI in 2022.

By comparison, closed-source tools are opaque with respect to their security features. It is harder for bad actors to generate illicit materials using these tools. But unlike open-source tools, closed-source tools are not subject to collective oversight because their inner workings cannot be examined by independent experts. Some closed-sourced models currently on the market include:

• GPT-4: A large multimodal model that accepts both image and text inputs and emits text outputs. Released by OpenAI in 2023.
• AlphaFold: Released by Google DeepMind in 2018, AlphaFold predicts protein structures from amino acid sequences.
• Copilot: A chatbot released by Microsoft in 2023.
• IBM Watson: A question-answering system developed by IBM to compete on Jeopardy in 2011, and released commercially in 2013.
• Siri: A digital assistant released by Apple in 2011.

1. Author’s statement. The author states that their primary purpose in introducing the bill is to establish a comprehensive regulatory framework to mitigate the harmful impacts of synthetic or “deep fake” content. According to the author, these include:
2. Harms caused by inauthentic content presented as authentic: The bill acknowledges the wide range of potential harms caused by inauthentic content, including financial scams, non-consensual intimate imagery, disinformation (especially around elections and public health), and the erosion of trust in the digital information ecosystem. The PAWS Act seeks to reduce these harms by requiring clear disclosure of content provenance, making it harder for inauthentic content to be mistaken as authentic.
3. Lack of transparency around provenance of digital media: The bill addresses transparency concerns by mandating that generative AI providers embed imperceptible and indelible watermarks containing provenance data in all synthetic content they create, and prominently display this provenance data to users. The bill also establishes standards for digital cameras and recording devices to offer watermarking options for authentic content.
4. Facilitation of harmful acts by bad actors: The bill prohibits the distribution of tools designed to remove watermarks or manipulate provenance data, making it more difficult for bad actors to generate unlabeled inauthentic content. It also requires conversational AI systems to disclose their artificial nature and obtain user consent, making it more difficult for bad actors to leverage AI generation for deception.

In sum, the bill seeks to establish clear standards and requirements around content provenance disclosure, watermarking, and labeling, with the goal of increasing transparency and reducing the ability of bad actors to deceive users with unlabeled synthetic content. By doing so, the author aims to mitigate the various potential harms enabled by increasingly sophisticated generative AI technologies.

1. The problem. Image manipulation and video doctoring have existed for nearly as long as photography and recording equipment, but they have historically required great effort and talent. In the past few years the rapid development of GenAI has drastically reduced those barriers to entry, allowing a vast quantity of convincing – but ultimately fake – content to be generated in an instant. The creation of text, imagery, video, and audio by GenAI has the potential to change the world by automating repetitive tasks and fostering creativity. When employed by bad actors, however, these capabilities have the potential to destroy lives and destabilize societies.

Nonconsensual pornography. GenAI has been used to create pornography since its inception. This content is inevitably nonconsensual, and as GenAI technology improves, these products will become harder to distinguish from reality. Women have always been the primary victims of these efforts; in the run-up to the 2024 Super Bowl, a series of images involving Taylor Swift began to appear on the social media platform X (formerly Twitter). These images were removed over the following days, but the damage had been done:

“We are too little, too late at this point, but we can still try to mitigate the disaster that’s emerging,” says Mary Anne Franks, a professor at George Washington University Law School and president of the Cyber Civil Rights Initiative. Women are “canaries in the coal mine” when it comes to the abuse of artificial intelligence, she adds. “It's not just going to be the 14-year-old girl or Taylor Swift. It’s going to be politicians. It’s going to be world leaders. It’s going to be elections.”^[1]

The harms of nonconsensual AI-powered pornography are already being felt in California:

A third school in Southern California has been hit with allegations of digitally manipulated images of students circulating around campus . . . “Sixteen eighth-grade students were identified as being victimized, as well as five egregiously involved eighth-grade students,” Superintendent Michael Bregy wrote. While Bregy acknowledged that children “are still learning and growing, and mistakes are part of the process,” he affirmed disciplinary measures had been taken and noted that the incident was swiftly contained. The district vowed to hold accountable any other students “found to be creating, disseminating, or in possession of AI-generated images of this nature.”^[2]

Scams. GenAI-powered speech and video is driving a new era in scamming. These AI tools are trained on publicly available data – the more data a target has online, the easier it is to develop a passable imitation of them or their loved ones. This is especially true of wealthy clients, whose public appearances, including speeches, are often widely available on the internet.^[3] For example, a complicated scam utilizing both deepfake video and false audio was recently performed in Hong Kong. A multinational company lost $25.6 million after employees were fooled by deepfake technology, with one incident involving a digitally recreated version of its chief financial officer ordering money transfers in a video conference call. Everyone present on the video call, except the victim, was a fake representation of real people. The scammers applied deepfake technology to turn publicly available video and other footage into convincing versions of the meeting’s participants.^[4]

Political propaganda and disinformation. Deepfake technology is being used around the world to spread disinformation and propaganda. 2024 is a major election year in democracies around the globe: at least 64 countries will hold elections, representing close to 49% of the world’s population.^[5] It is also likely to be the first of many election years in which AI plays a pivotal role, as the technology becomes more widely available and easier to use. This has already been observed in Slovakia, where deepfake audio influenced an election in 2023:

Days before a pivotal election in Slovakia to determine who would lead the country, a damning audio recording spread online in which one of the top candidates seemingly boasted about how he’d rigged the election. And if that wasn’t bad enough, his voice could be heard on another recording talking about raising the cost of beer. The recordings immediately went viral on social media, and the candidate, who is pro-NATO and aligned with Western interests, was defeated in September by an opponent who supported closer ties to Moscow and Russian President Vladimir Putin.^[6]

Similar deepfakes have now been deployed in the United States in advance of the 2024 presidential election. In late January, between 5000 and 20,000 New Hampshire residents received AI-generated phone calls impersonating President Biden that told them not to vote in the state’s primary. The call told voters: “It’s important that you save your vote for the November election.” Concern about this call has led at least 14 states to introduce legislation targeting AI-powered disinformation.^[7] It is still unclear how many people might not have voted based on these calls.

Deepfakes are not only being deployed by third parties; they can be used by the candidates themselves, either to improve their own self-images or to detract from their opponents. In mid-2023, former Republican presidential candidate Governor Ron DeSantis used AI to add fighter jets to one of his campaign videos.^[8] Around the same time, Governor DeSantis’ super PAC released an ad containing an AI-generated speech by former president Donald Trump.^[9] The Republican National Committee also released a 30-second ad that displayed images of disorder and destruction, with a voiceover that described the “consequences” of re-electing President Biden.^[10] None of the images in this ad were real.

1. The solution. In principle, the issue of content authenticity can be solved by carrying out a three-step process:
2. Require all content produced by GenAI to be labeled “fake.”
3. Require all content produced by cameras and other recording devices to be labeled “real.”
4. Require all online platforms to prominently display these labels.

This is the precise strategy adopted by the bill. But is such a solution feasible? The devil is, of course, in the details. Each step in this process has a variety of technical and legal hurdles to overcome – but there is no real alternative. It is not currently possible to develop a tool that perfectly identifies AI-generated content. And as GenAI technology continues to develop, this situation will only become more dire.

1. Step one. Step one of the three-step content authenticity plan outlined above requires all content produced by GenAI to be labeled “fake.” To accomplish this, AB 3211 would require what it calls “generative AI system providers” to build watermarking functionality into the GenAI systems released by those providers. Watermarking refers to the practice of embedding visible or invisible markers into GenAI products. This is a tricky technology – at present, there is no single set of industry standards that system providers can turn to as they grapple with this bill’s requirements. Conversations between committee staff and the author have revealed that this is partially by design: in requiring a good-faith effort to adopt watermarking technologies on the part of GenAI providers, the author hopes to incentivize the development of “best practices” for watermarking.

The bill defines “generative AI system providers” to encompass the organizations and individuals that make GenAI systems – or substantial components thereof, such as the parameters that make these systems tick – available, or that provide them as standalone models, or that embed them into other systems. So far, so good: each of these entities is capable of building watermarking functionality into their products. However the definition goes on to include individuals and organizations that put these systems into service (often termed “deployers”), as well as online repositories or hosting websites that make systems available “even if those repositories are not the original makers of the AI systems they make available.” It is not clear how these parties could abide by the other requirements of this bill, given that they do not have the ability to edit the GenAI systems they work with. By including them in this definition, they are effectively excluded from the market by virtue of being unable to comply. A committee amendment has been proposed to narrow this definition such that it specifically targets parties who can carry out the provisions of this bill.

Watermarking is a tricky technology. Watermarks can be stripped from content relatively easily by common screenshotting tools, file compression software, and image editing programs like Photoshop. They can also be faked by treating a GenAI system like a copy machine: a real image or video can be fed into a GenAI system and spit back out, unaltered except for the addition of a watermark. The system’s user receives a piece of authentic content that has been marked as inauthentic, ready to be posted online in order to create confusion and sow discord.

AB 3211 accounts for these inherent weaknesses in two ways: first, by requiring that providers and other entities named throughout this bill make their tools and systems “compatible with widely used industry standards.” Second, this bill requires GenAI providers to conduct “AI red-teaming exercises” in collaboration with third-party experts. These red-teaming exercises are described in the White House’s recent executive order on AI:

The term “AI red-teaming” means a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI.  Artificial Intelligence red-teaming is most often performed by dedicated “red teams” that adopt adversarial methods to identify flaws and vulnerabilities, such as harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associated with the misuse of the system.^[11]

Upon completing a red-teaming exercise, a provider is required to both post the results of the exercise online and submit the results to CDT. Upon discovering a vulnerability or flaw in their system, a provider is required to submit a report to CDT, inform all GenAI system providers who might be affected by similar vulnerabilities or flaws, and inform all affected users. This third point presents a privacy concern: in order to inform affected users, this bill would require a provider to collect the contact information of its users. A committee amendment has been proposed that would limit the requirement to inform users to those users whose contact information the provider has already collected, thus removing the incentive for providers to collect contact information from all users.

This bill imposes a third and final requirement on GenAI system providers: providers must develop software that allows a user to verify whether any given piece of content was created by the provider’s system. The bill refers to this software as a “watermark decoder.” This is by far the most difficult requirement imposed on providers of GenAI systems. The bill specifically requires that this software be capable of being used “in an efficient and automated fashion by online platforms or researchers seeking to assess the provenance of hundreds or thousands of pieces of content per minute.” The costs associated with this requirement would likely be exorbitant, effectively limiting the release of GenAI systems to large, well-established entities. A committee amendment has been introduced to target the use of these tools at users seeking to test individual pieces of content, rather than high-throughput testing by large online platforms. Watermarking decoders have another issue: their existence makes it substantially easier to intentionally strip watermarks out of labeled content. A piece of labeled content can be slightly altered and passed through a watermarking decoder to have the effect of the alteration read out. By repeating this process, a bad actor can determine exactly how much a piece of inauthentic content must be altered before a watermarking decoder no longer recognizes it as inauthentic.

This bill would also prohibit the release of GenAI systems whose watermarking functionalities can be removed, effectively banning the release of open-source GenAI systems in California. This could severely disadvantage California’s GenAI industry: two of the world’s largest open-source GenAI systems are Meta’s Llama2, based in California, and Mistral AI’s Mistral 7B, based in France. The effect of this provision would likely be to remove Llama2 from contention. Banning the use of open-source GenAI would also severely impact the small business community, as training GenAI from scratch is extremely expensive and time intensive. Many small businesses build their tools and systems by using open-source GenAI systems as a starting point. Given that the permissibility of open-source AI remains a contentious, open question, a committee amendment has been proposed to strike this provision from the bill.

GenAI providers offering conversational AI systems are required to disclose their nature to users. In visual interfaces, the disclosure must be visible throughout the interaction. In audio interfaces, the disclosure must be made at the beginning and end of the interaction. In all cases, the conversational AI is required to obtain affirmative consent from the user before proceeding.

1. Step two. Step two of the three-step content authenticity plan requires all “real” content to be labeled as such. This bill would require newly manufactured digital cameras and other recording devices sold or distributed in California to offer users the option to place watermarks in recorded content. Manufacturers of existing cameras are required, if possible, to provide updates that grant watermarking functionality.
2. Step three. The third and final step of the three-step content authenticity plan requires large online platforms to use labels to prominently display provenance data found in watermarks that are, by now, embedded in both real and GenAI-produced content. “Large online platforms” are defined to include websites, web applications and digital applications that have at least one million California users per year. These platforms are additionally required to use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed, or that was produced by AI systems without watermarking functionalities.

AB 3211 requires large online platforms to warn their users that failing to disclose GenAI-produced content is not permissible, and will result in disciplinary action. The bill goes on to require these platforms to disable the accounts of users who upload GenAI-produced content without disclosing it. Because this raises substantial First Amendment concerns, a committee amendment has been proposed to remove it from the bill.

1. Additional considerations. While the bill, as proposed to be amended, has been substantially clarified and refined, there are a number of considerations that the author may wish to address going forward:

Artificial text and audio. Watermarking, as a technology, is in its infancy. Progress has been made towards developing standards for watermarking of images and video, but how audio and text should be watermarked is far less clear. This bill does not discriminate between these types of content, despite drastic differences in the feasibility of incorporating watermarks into each.

Bananas and bite-sized content. “Invisible and maximally indelible” watermarks usually take the form of statistical correlations between different parts of a GenAI product: for example, the intensities of certain pixels in an image (if the GenAI product is visual) or the choice of words in paragraph (if the GenAI product is textual). The inclusion of watermarks inevitably degrades the quality of a GenAI product, and this effect becomes more profound as the GenAI product becomes smaller. For example: imagine a textual GenAI product that is watermarked by including the word “banana” 10 times throughout the document. If the document is tens of thousands of words long, the word “banana” does not stand out. But if the document is a short snippet of 100 words, “banana” appears constantly. The document would contain almost nothing else. The smaller a GenAI product is, the more it must be altered from its ideal form in order to include a watermark.

The Department of Technology? The bill tasks the CDT with administrative enforcement of violations. CDT is under the Government Operations Agency and is organized into several offices. CDT oversees nearly all aspects of information technology (IT) in state government, including advising the Governor on the strategic management and direction of the State’s IT resources; issuing and maintain policies, standards, and procedures governing the State's information security; and providing oversight of the State’s IT projects. Thus, it appears that CDT is generally tasked with managing the State’s internal operations rather than actively regulating private industry. The author may wish to explore whether CDT is the appropriate enforcement entity.

“One Million Dollars.” The bill authorizes CDT to assess an administrative enforcement of up to $1 million or five percent of the violator’s annual global revenue, whichever is greater. In 2023, for example, Meta generated $134 billion in revenue. A single violation of this bill could have resulted in a $6.7 billion fine. The author may wish to set a ceiling on the penalty that is more proportionate to the harm of a violation.

Compelled speech. With respect to conversational AI systems and provenance data in watermarks, the bill requires certain disclosures and thus compels speech. Because the right to speak encompasses the right not to speak, these provisions implicate the First Amendment. Should this bill pass this Committee, these issues will be more thoroughly examined in the Judiciary Committee.

8) Committee amendments. In addition to the substantive amendments described above, the author, sponsors, and committee staff worked together to revise and recast the bill and make a number of clarifying and technical change. As proposed to be amended, the bill’s language, in its entirety, would be replaced with the following:

SECTION 1. The Legislature finds and declares all of the following:

(a) Generative artificial intelligence (GenAI) technologies are increasingly able to produce inauthentic images, audio, video, and text content, in ways that are harmful to society.

(b) In order to reduce the severity of the harms caused by GenAI, it is important for GenAI content to be clearly disclosed and labeled.

(c) Failing to appropriately label GenAI content can skew election results, enable academic dishonesty, and erode trust in the online information ecosystem.

(d) The Legislature should act to adopt standards pertaining to the clear disclosure and labeling of GenAI content, in order to alleviate harms caused by the misuse of these technologies.

(e) The Legislature should push for the creation of tools that allow Californians to assess the authenticity of online content.

(f) The Legislature should require online platforms to label inauthentic content produced by GenAI.

(g) Through these actions, the Legislature can help to ensure that Californians remain safe and informed.

SEC. 2. Chapter 41 (commencing with Section 22949.90) is added to Division 8 of the Business and Professions Code, to read:

CHAPTER 41. California Provenance, Authenticity and Watermarking Standards

22949.90. For purposes of this chapter, the following definitions apply:

(a) “AI red-teaming” means a structured testing effort to find flaws and vulnerabilities in an AI system, including, but not limited to, harmful or discriminatory outputs, unforeseen or undesirable system behaviors, limitations, or potential risks associated with misuse of the system.

(b) “Artificial intelligence” or “AI” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.

(c) “Authentic content” means images, videos, audio, or text created by human beings without any modifications or with only minor modifications that do not lead to significant changes to the perceived contents or meaning of the content. Minor modifications include, but are not limited to, changes to brightness or contrast of images, removal of background noise in audio, and spelling or grammar corrections in text.

(d) “Conversational AI systems” means chatbots and other audio- or video-based systems that can hold humanlike conversations through digital media, including, but not limited to, online calling, phone calling, video conferencing, messaging, application or web-based chat interfaces, or other conversational interfaces. Conversational AI systems include, but are not limited to, chatbots for customer service or entertainment purposes embedded in internet websites and applications.

(e) “Digital signature” means a digital method that allows a user to sign a piece of authentic or synthetic content with their name or device information, verifying that they created the content.

(f) “Generative AI system” means an artificial intelligence system that generates derived synthetic content, including images, videos, audio, text, and other digital content.

(g) “Generative AI provider” means an organization or individual that creates, codes, substantially modifies or otherwise produces a generative AI system.

(h) “Generative AI hosting platform” means an online repository or other internet website that makes generative AI systems available for download.

(i) “Inauthentic content” means synthetic content that is so similar to authentic content that it could be mistaken as authentic.

(j) “Large online platform” means a public-facing internet website, web application, or digital application, including a social network, video sharing platform, messaging platform, advertising network, or search engine that had at least 1,000,000 California users during the preceding 12 months.

(k) “Maximally indelible watermark” means a watermark that is designed to be as difficult to remove as possible using state-of-the-art techniques and relevant industry standards.

(l) “Provenance data” means data that identifies the origins of synthetic content, including but not limited to:

1. the name of the generative AI provider.
2. the name and version number of the AI system that generated the content.
3. the time and date of the creation.
4. which portions of the content are synthetic.

(m) “Synthetic content” means information, including images, videos, audio, and text, that has been produced or significantly modified by a generative AI system.

(n) “Watermark” means information that is embedded into a generative AI system’s output for the purpose of conveying its synthetic nature, identity, provenance, history of modifications, or history of conveyance.

(o) “Watermark decoders” means freely available software tools or online services that can read or interpret watermarks and output the provenance data embedded in them.

22949.90.1. (a) A generative AI provider shall do all of the following:

1. Place imperceptible and maximally indelible watermarks containing provenance data into synthetic content produced or significantly modified by a generative AI system that the provider makes available.
2. If a sample of synthetic content is too small to contain the required provenance data, the provider shall, at minimum, attempt to embed watermarking information that identifies the content as synthetic and provide the following provenance information in order of priority, with subsection (i) being the most important, and subsection (iv) being the least important:
   1. The name of the generative AI provider.
   2. The name and version number of the AI system that generated the content.
   3. The time and date of the creation of the content.
   4. If applicable, which specific portions of the content are synthetic.
3. To the greatest extent possible, watermarks shall be designed to retain information that identifies content as synthetic and gives the name of the provider in the event that a sample of synthetic content is corrupted, downscaled, cropped, or otherwise damaged.

(2) Develop downloadable watermark decoders that allow a user to determine whether a piece of content was created with the provider’s system, and make those tools available to the public.

(A) The watermark decoders shall be easy to use by individuals seeking to quickly assess the provenance of a single piece of content.

(B) The decoders shall adhere, to the greatest extent possible, to relevant national or international standards.

(3) Conduct AI red-teaming exercises involving third-party experts to test whether watermarks can be easily removed from synthetic content produced by the provider’s generative AI systems, as well as whether the provider’s generative AI systems can be used to falsely add watermarks to otherwise authentic content. Red-teaming exercises shall be conducted before the release of any new generative AI system and annually thereafter.

(A) If a provider allows their generative AI systems to be downloaded and modified, the provider shall additionally conduct AI red-teaming to assess whether their systems’ watermarking functionalities can be disabled.

(B) A provider shall make summaries of its AI red-teaming exercises publicly available in a location linked from the home page of the provider’s internet website, using a clearly labeled link that has a similar look, feel, and size relative to other links on the same web page. The provider shall remove from the summaries any details that pose an immediate risk to public safety.

(C) A provider shall submit full reports of its AI red-teaming exercises to the Department of Technology within six months of conducting a red-teaming exercise pursuant to this section.

(b) A generative AI provider may continue to make available a generative AI system that was made available before the date upon which this act takes effect, and that does not have watermarking capabilities as described by paragraph (1) of subdivision (a), if either of the following conditions are met:

(1) The provider is able to retroactively create and make publicly available a decoder that accurately determines whether a given piece of content was produced by the provider’s system with at least 99 percent accuracy as measured by an independent auditor.

(2) The provider conducts and publishes research to definitively demonstrate that the system is not capable of producing inauthentic content.

(c) Providers and distributors of software and online services shall not make available a system, application, tool, or service that is designed to remove watermarks from synthetic content.

(d) Generative AI hosting platforms shall not make available a generative AI system that does not place maximally indelible watermarks containing provenance data into content created by the system.

(f) (1) Within 24 hours of discovering a vulnerability or failure in a generative AI system, a generative AI provider shall report the vulnerability or failure to the Department of Technology.

(A) A provider shall notify other generative AI providers that may be affected by similar vulnerabilities or failures in a manner that allows the other generative AI system provider to harden their own AI systems against similar risks.

(B) A provider shall notify affected parties, including, but not limited to, online platforms, researchers or users who received incorrect results from a watermark decoder, or users who produced AI content that contained incorrect or insufficient watermarking data. A provider shall not be required to notify an affected party whose contact information the provider has not previously collected or retained.

(2) A provider shall make any report to the Department of Technology under this subdivision publicly available in a location linked from the home page of the provider’s internet website with a clearly labeled link that has a similar look, feel, and size relative to other links on the same web page. If public disclosure of the report under this subdivision could pose public safety risks, a provider may instead do either of the following:

(A) Post a summary disclosure of the reported vulnerability or failure.

(B) Delay, for no longer than 30 days, the public disclosure of the report until the public safety risks have been mitigated. If a provider delays public disclosure, they shall document their efforts to resolve the vulnerability or failure as quickly as possible in order to meet the reporting requirements under this subdivision.

(g) (1) A conversational AI system shall clearly and prominently disclose to users that the conversational AI system generates synthetic content.

(A) In visual interfaces, including, but not limited to, text chats or video calling, a conversational AI system shall place the disclosure required under this subdivision in the interface itself and maintain the disclosure’s visibility in a prominent location throughout any interaction with the interface.

(B) In audio-only interfaces, including, but not limited to, phone or other voice calling systems, a conversational AI system shall verbally make the disclosure required under this subdivision at the beginning and end of a call.

(2) In all conversational interfaces of a conversational AI system, the conversational AI system shall, at the beginning of a user’s interaction with the system, obtain a user’s affirmative consent acknowledging that the user has been informed that they are interacting with a conversational AI system. A conversational AI system shall obtain a user’s affirmative consent prior to beginning the conversation.

(3) Disclosures and affirmative consent opportunities shall be made available to a user in the language in which the conversational AI system is communicating with the user.

(4) The requirements under this subdivision shall not apply to conversational AI systems that do not produce inauthentic content.

(h) This section shall become operative on February 1, 2025.

22949.90.2. (a) For purposes of this section, the following definitions apply:

(1) “Authenticity watermark” means a watermark of authentic content that includes the name of the device manufacturer.

(2) “Camera and recording device manufacturers” means the makers of a device that can record photographs, audio, or video content, including, but not limited to, video and still photography cameras, mobile phones with built-in cameras or microphones, and voice recorders.

(3) “Provenance watermark” means a watermark of authentic content that includes details about the content, including, but not limited to, the time and date of production, the name of the user, details about the device, and a digital signature.

(b) (1) Beginning January 1, 2026, newly manufactured digital cameras and recording devices sold, offered for sale, or distributed in California shall offer users the option to place an authenticity watermark and provenance watermark in the content produced by that device.

(2) A user shall have the option to remove the authenticity and provenance watermarks from the content produced by their device.

(3) Authenticity watermarks shall be turned on by default, while provenance watermarks shall be turned off by default.

(4) Newly manufactured digital cameras and recording devices subject to the requirements of this subdivision shall clearly inform a user of the existence of the authenticity and provenance watermarks settings upon the user’s first use of the camera or the recording function on the recording device.

(A) When a camera or audio recording application is open, a newly manufactured digital camera or recording device shall have a clear indicator that a watermark is being applied.

(B) A newly manufactured digital camera or recording device shall allow the user to adjust the watermarks settings.

(5) Authenticity and provenance watermarks shall, if enabled, be applied to authentic content produced using third-party applications that bypass default camera or recording applications in order to offer camera or audio recording functionalities.

(c) The authenticity watermark and provenance watermark, as required in subdivision (b), shall be compatible with widely used industry standards

(d) Beginning January 1, 2026, a camera and recording device manufacturer shall offer a software or firmware update enabling a user to place an authenticity watermark and provenance watermark on the content created by the device to a user of a digital camera or recording device purchased in California prior to January 1, 2026, if technically feasible.

22949.90.3. (a) Beginning March 1, 2025, a large online platform shall use labels to prominently disclose the provenance data found in watermarks or digital signatures in content distributed to users on its platforms.

(1) The labels shall indicate whether content is fully synthetic, partially synthetic, authentic, authentic with minor modifications, or does not contain a watermark.

(2) A user shall be able to click or tap on a label to inspect provenance data in an easy-to-understand format.

(b) The disclosure required under subdivision (a) shall be readily legible to an average viewer or, if the content is in audio format, shall be clearly audible. A disclosure in audio content shall occur at the beginning and end of a piece of content and shall be presented in a prominent manner and at a comparable volume and speaking cadence as other spoken words in the content. A disclosure in video content should be legible for the full duration of the video.

(c) A large online platform shall use state-of-the-art techniques to detect and label synthetic content that has had watermarks removed, or that was produced by generative AI systems without watermarking functionality.

(d) (1) A large online platform shall require a user that uploads or distributes content on its platform to disclose whether the content is synthetic content.

(2) A large online platform shall include prominent warnings to users that upload or distribute synthetic content without disclosing that it is synthetic content may result in disciplinary action.

(3) A large online platform may provide users with an option to indicate that the user is uncertain whether the content they are uploading or distributing is synthetic content. If a user uploads or distributes content and indicates that they are uncertain of whether the content is synthetic content, a large online platform shall indicate that the uploaded or distributed content is possibly synthetic and shall display that indication in a manner that is visible or audible to viewers or listeners of the content.

(e) A large online platform shall use state-of-the-art techniques to detect and label text-based inauthentic content that is uploaded by users.

(f) A large online platform shall make accessible a verification process for users to apply a digital signature to authentic content. The verification process shall include options that do not require disclosure of personal identifiable information.

22949.90.4. (a) (1) Beginning January 1, 2026, and annually thereafter, generative AI providers and large online platforms shall produce a Risk Assessment and Mitigation Report that assesses the risks posed and harms caused by synthetic content generated by their systems or hosted on their platforms.

(2) The report shall include, but not be limited to, assessments of the distribution of AI-generated child sexual abuse materials, nonconsensual intimate imagery, disinformation related to elections or public health, plagiarism, or other instances where synthetic or inauthentic content caused or may have the potential to cause harm.

(3) The report shall be audited by qualified, independent auditors who shall assess and either validate or invalidate the claims made in the report. Auditors shall use state-of-the-art techniques to assess reports, and shall adhere to relevant national and international standards.

22949.90.5. A violation of this chapter may result in an administrative penalty, assessed by the Department of Technology, of up to one million dollars ($1,000,000) or 5 percent of the violator’s annual global revenue, whichever is greater.

22949.90.6. Within 90 days of the date upon which this act takes effect, the Department of Technology shall adopt regulations to implement and carry out the purposes of this chapter. The agency shall review and update its regulations relating to the implementation of this chapter as needed, including, but not limited to, adopting specific national or international standards for provenance, authenticity, watermarking, and digital signatures, so long as the standards do not weaken the provisions of this chapter.

22949.91. The provisions of this chapter are severable. If any provision of this chapter or its application is held invalid, that invalidity shall not affect other provisions or applications that can be given effect without the invalid provision or application.

9) Related legislation. AB 1791 (Weber, 2024) prohibits a social media platform from removing digital content provenance verification from content uploaded to the social media platform by a user. The bill is pending in this Committee.

AB 3050 (Low, 2024) would require CDT to issue regulations to establish standards for watermarks to be included in covered AI-generated material. The bill is pending in this Committee.

ARGUMENTS IN SUPPORT

The sponsor, CITED, writes:

California has the opportunity to lead the U.S. response to these threats. As the home of Silicon Valley and the heartland of AI innovation, California can and must play an important role in shaping the solution. To that end, AB 3211 represents a phased-in solution to address this complex problem that complements existing standard-setting efforts. The bill has been developed with insights from EU regulators who have worked intimately on the AI Act and from federal policy experts who see, in the face of congressional inaction, California’s power to drive nationwide change.

California Voter Foundation writes:

California lawmakers have a long history and strong track record of ensuring voters can make informed choices through enhanced campaign finance disclosure laws, truth in committee naming laws, and requiring counties to list supporters for and against ballot measures directly on California ballots. AB 3211 continues California lawmakers’ longstanding efforts to empower California voters to make informed, confident choices by enabling the public to verify the authenticity of media information.

Courage California writes:

As new technologies propel us into the future, we must move forward in a way that maximizes transparency, mitigates harm, and protects users. AB 3211, which borrows heavily from the guidance and leadership of the European Union’s AI Act, accomplishes these goals.

ARGUMENTS IN OPPOSITION

A coalition of industry associations including CalChamber, Netchoice, Technet, and the Computer & Communications Industry Association writes:

AB 3211 seems to treat all AI-generated content as inherently bad or risky. By requiring such thorough and prescriptive requirements for content labeling, the bill makes a value judgment that consumers must be notified and aware of any content that was created by AI. This applies to all inauthentic content including purely artistic or satirical content. While we agree with the intent to provide more information to consumers, in some instances it could create disclosure or notification fatigue. If watermarks and content credentials become so routine and placed on all AI-generated content, users may start to ignore and disregard their presence. Rather than focusing on whether the content itself was AI-generated, synthetic, or inauthentic, we would advise focusing on the misuse of this technology.

REGISTERED SUPPORT / OPPOSITION:

Support

California Initiative for Technology & Democracy, a Project of California Common CAUSE (CITED) (sponsor)

Asian Americans and Pacific Islanders for Civic Empowerment
Asian Law Alliance
California Initiative for Technology & Democracy, a Project of California Common CAUSE
California Voter Foundation
Courage California

Support If Amended

Oakland Privacy

Opposition

California Chamber of Commerce
Computer & Communications Industry Association
Netchoice
Technet

Analysis Prepared by: Slater Sharp and Josh Tosney / P. & C.P. / (916) 319-2200

1. Brian Contreras, "Tougher AI Policies Could Protect Taylor Swift—And Everyone Else—From Deepfakes," Feb. 8. 2024, www.scientificamerican.com/article/tougher-ai-policies-could-protect-taylor-swift-and-everyone-else-from-deepfakes/. ↑

2. Mackenzie Tatananni, “'Inappropriate images' circulate at yet another California high school, as officials grapple with how to protect teens from AI porn created by classmates,” Daily Mail, Apr. 11, 2024, https://www.dailymail.co.uk/news/article-13295475/Inappropriate-images-California-Fairfax-High-School-AI-deepfake.html ↑

3. Emily Flitter and Stacy Cowley, "Voice Deepfakes Are Coming for Your Bank Balance", New York Times, Aug. 30, 2023, www.nytimes.com/2023/08/30/business/voice-deepfakes-bank-scams.html. ↑

4. Harvey Kong, "‘Everyone looked real’: multinational firm’s Hong Kong office loses HK$200 million after scammers stage deepfake video meeting," South China Morning Post, Feb. 4, 2024, www.scmp.com/news/hong-kong/law-and-crime/article/3250851/everyone-looked-real-multinational-firms-hong-kong-office-loses-hk200-million-after-scammers-stage. ↑

5. Koh Ewe, "The Ultimate Election Year: All the Elections Around the World in 2024," Time, Dec. 28, 2023, https://time.com/6550920/world-elections-2024/. ↑

6. Curt Devine, Donie O'Sullivan, Sean Lyngass, "A fake recording of a candidate saying he’d rigged the election went viral. Experts say it’s only the beginning," CNN, Feb. 1, 2024, www.cnn.com/2024/02/01/politics/election-deepfake-threats-invs/index.html. ↑

7. Adam Edelman, "States turn their attention to regulating AI and deepfakes as 2024 kicks off," NBC News, Jan. 22, 2024, www.nbcnews.com/politics/states-turn-attention-regulating-ai-deepfakes-2024-rcna135122. ↑

8. Ana Faguy, "New DeSantis Ad Superimposes Fighter Jets In AI-Altered Video Of Speech," Forbes, May 25, 2023, www.forbes.com/sites/anafaguy/2023/05/25/new-desantis-ad-superimposes-fighter-jets-in-ai-altered-video-of-speech/. ↑

9. Alex Isenstadt, "DeSantis PAC uses AI-generated Trump voice in ad attacking ex-president," Politico, Jul. 17, 2023, www.politico.com/news/2023/07/17/desantis-pac-ai-generated-trump-in-ad-00106695. ↑

10. GOP, "Beat Biden," Apr. 25, 2023, https://www.youtube.com/watch?v=kLMMxgtxQ1Y. ↑

11. “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” White House, Oct. 30, 2023, https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/. ↑