Existing law requires the Superintendent of Public Instruction to establish procedures within the State Department of Education to, among other things, annually identify the critical needs for which effective educational programs and practices are to be identified, developed, and disseminated to public schools.

This bill would require the Superintendent, in consultation with the State Board of Education, to convene a working group, composed as provided, for specific purposes related to artificial intelligence in public schools, as specified. The bill would require, among other things, the working group to develop, on or before January 1, 2026, guidance for local educational agencies and charter schools on the safe use of artificial intelligence in education, and to, on or before July 1, 2026, develop a model policy for local educational agencies and charter schools regarding the safe and effective use of artificial intelligence in ways that benefit, and do not harm, pupils and educators, as provided. The bill would require the working group to, on or before September 1, 2026, report its findings and recommendations to the appropriate policy and fiscal committees of the Legislature, the Legislative Analysts Office, the state board, and the Department of Finance, as provided. The bill would make implementation of these provisions contingent upon an appropriation by the Legislature, as provided.

Existing law requires the Superintendent of Public Instruction to establish procedures within the State Department of Education to, among other things, annually identify the critical needs for which effective educational programs and practices are to be identified, developed, and disseminated to public schools.

This bill would require the Superintendent, in consultation with the State Board of Education, to, on or before January 1, 2025, convene a working group for the purpose of exploring how artificial intelligence and other forms of similarly advanced technology are currently being used in education, identifying how they may be used in the future, and developing best practices to ensure that those technologies advance, rather than harm, educational quality, pupil critical thinking and writing skills, and the essential work of certificated and classified employees.

Date of Hearing: May 8, 2024

ASSEMBLY COMMITTEE ON APPROPRIATIONS

Buffy Wicks, Chair

ABPCA Bill Id:AB 2652

Author:(Muratsuchi) – As Amended Ver:April 18, 2024

Urgency: No State Mandated Local Program: No Reimbursable: No

SUMMARY:

This bill requires, subject to appropriation, the Superintendent of Public Instruction to convene a workgroup related to artificial intelligence (AI) in educational settings.

Specifically, this bill requires the workgroup, by January 1, 2026, to develop guidance for local educational agencies on the safe use of AI in education. The bill further requires, by July 1, 2026, the workgroup develop a model policy for local educational agencies regarding the safe and effective use of AI. The bill further requires, by September 1, 2026, the workgroup submit a report to the Legislature and others on the process and products of the workgroup and any related findings or recommendations.

FISCAL EFFECT:

One-time General Fund costs to the California Department of Education (CDE) of between $250,000 and $500,000 for CDE to coordinate and convene the workgroup, assuming up to 20 members who would not receive compensation and the workgroup would have six meetings.

COMMENTS:

1. Purpose. According to the author:

Artificial intelligence has the potential to assist teachers and enhance quality of education. It also has the potential to enable plagiarism, reduce critical thinking skills and displace essential workers. AB 2652 directs the Superintendent of Public Instruction to convene a working group comprised of various stakeholders to thoughtfully address how schools can best integrate this new technology so that it works with us, not against us.

1. Related Legislation. AB 2876 (Berman) requires the Instructional Quality Commission to consider incorporating AI literacy content into the mathematics, science, and history-social science curriculum frameworks when those frameworks are next revised; and to ensure that the mathematics, science, and history-social science instructional materials that it recommends to the state board for adoption after January 1, 2025, include AI literacy content. The bill is pending on the Assembly floor.

SB 721 (Becker) establishes the California Interagency AI Working Group, to deliver a report regarding AI to the Legislature. The bill is pending in the Assembly Privacy and Consumer Protection Committee.

SB 1235 (Gonzalez) requires a public institution of higher education to establish the AI and Deepfake Working Group, to evaluate and advise the Legislature and the public on the relevant issues and impacts of AI and deepfakes. The bill is pending in the Senate Education Committee.

Analysis Prepared by: Natasha Collins / APPR. / (916) 319-2081

Date of Hearing: April 16, 2024

ASSEMBLY COMMITTEE ON PRIVACY AND CONSUMER PROTECTION

Rebecca Bauer-Kahan, Chair

ABPCA Bill Id:AB 2652 (

Author:Muratsuchi) – As Amended Ver:April 8, 2024

AS PROPOSED TO BE AMENDED

PROPOSED CONSENT

SUBJECT: State Department of Education: artificial intelligence working group

SYNOPSIS

This bill directs the Superintendent of Public Instruction to convene a working group in order to study the risks and benefits of artificial intelligence (AI) in education. The working group would be composed of teachers, public school staff, administrators, university and community college faculty, private sector representatives, and pupils enrolled in public school. In assessing the current and future state of AI use in education, the working group would focus especially on the risks and benefits of AI for students and teachers.

This bill is sponsored by the California Federation of Teachers and supported by a number of California teacher and education employee associations, including California Association of School Business Officials, California School Employees Association, and the California Teachers Association. It is also supported by the Los Angeles County Office of Education. The bill has no opposition.

This bill passed the Assembly Education Committee on a 7-0 vote.

SUMMARY: Requires the Superintendent of Public Instruction (SPI), in consultation with the State Board of Education (SBE), to convene a working group for the purpose of exploring how artificial intelligence (AI) and other forms of similarly advanced technology are currently being used in education. Specifically, this bill:

1. Requires the SPI, on or before January 1, 2025, in consultation with the SBE, to convene a working group for the purposes of:
2. Exploring how AI is currently being used in education.
3. Identifying how it may be used in the future.
4. Developing best practices to ensure that AI technologies advance, rather than harm, educational quality, pupil critical thinking and writing skills, and the essential work of certificated and classified employees.

EXISTING LAW:

1. Requires the Instructional Quality Commission (IQC) to consider developing and recommending to the SBE, on or before July 31, 2019, computer science content standards for kindergarten through 12th grade pursuant to recommendations developed by a group of computer science experts. (Ed. Code § 60605.4.)
2. States that if a school district requires more than two courses in mathematics for graduation from high school, the district may award a student up to one mathematics course credit for successfully completing a “category C” approved computer science course (Ed. Code § 51225.35.)
3. Requires the California State University (CSU), and requests the University of California (UC), to develop guidelines for high school computer science courses that may be approved for the purposes of recognition for admission (Ed. Code § 66205.5.)
4. Through regulation, authorizes holders of credentials in mathematics, business, and Industrial and Technology Education (ITE), as well as holders of supplementary authorizations in computer science, to teach computer science (Cal. Code Regs, Tit. 5, § 80005.)

FISCAL EFFECT: As currently in print, this bill is keyed fiscal.

COMMENTS:

1) Background. The development of AI is creating exciting opportunities to improve California’s education system – but with novel technologies come novel safety concerns. As AI becomes ubiquitous in schools, can students be confident that their personal information is not being captured and sold without their consent? Are the automated decision tools influencing the careers of educators being trained on unbiased datasets? When school districts partner with industry to develop AI tools for their classrooms, who ultimately owns these models and their outputs? The widespread adoption of AI has the potential to transform education for the better – but without careful implementation, these changes could also erode the privacy of students and educators while widening existing equity gaps.

Crash course on AI. AI uses algorithms – sets of rules – to transform inputs into outputs. Inputs and outputs can be anything a computer can process: numbers, text, audio, video, or movement. This is because AI is not fundamentally different from other computer functions. Its novelty lies in its application: unlike normal computer functions, AI is able to accomplish tasks that are normally performed by humans.

AI that are trained on small, specific datasets in order to make recommendations and predictions are sometimes referred to as “predictive AI.” This differentiates them from “generative AI,” (GenAI) which are trained on massive datasets in order to produce detailed text and images. When Netflix suggests a TV show to a viewer, the recommendation is produced by predictive AI that has been trained on the viewing habits of Netflix users.^[1] When ChatGPT generates text in clear, concise paragraphs, it uses GenAI that has been trained on the written contents of the internet.^[2]

AI is already integral to many aspects of modern society, and the advent of GenAI will undoubtedly lead to an even greater number of applications. However, AI is not an inherently benevolent or harmful technology – it is simply a tool.

Bias and bullying. There is a common saying in computer science: “garbage in, garbage out.” The performance of an AI tool is directly impacted by the quality, quantity, and relevance of the data used to train it.^[3] If the data used to train an AI is biased, the AI’s outputs will be similarly biased.

The capacity of AI to amplify existing biases can have major consequences, including on the careers of educators. People of various races, genders, and cultures are not distributed equally throughout the workforce. An AI trained on historical data to make hiring decisions will be predisposed to maintain the ratios it is trained on; as described by Aditya Malik, the Founder and CEO of Valuematrix.ai:

Generative AI, for all its grandeur, has the potential to perpetuate latent biases inherited from human creators. A disconcerting echo of historical prejudices may inadvertently seep into the algorithms. Imagine a scenario where previous senior managers, driven by biases of gender, age, faith or race, rejected candidates for misguided reasons. The AI, if not vigilantly curated, might misconstrue these patterns as indicators of incompetence, thus exacerbating the exclusion of qualified candidates from underrepresented backgrounds.^[4]

The interactions between humans and AI can themselves be subject to bias. If a voice-recognition system is trained using audio data that excludes certain accents, it can fail to understand accented speech once deployed:

“Clow-dia,” I say once. Twice. A third time. Defeated, I say the Americanized version of my name: “Claw-dee-ah.” Finally, Siri recognizes it. Having to adapt our way of speaking to interact with speech recognition technologies is a familiar experience for people whose first language is not English or who do not have conventionally American-sounding names. I have even stopped using Siri because of it.^[5]

Plagiarism. The originality of GenAI work products is frequently called into question. Tools such as ChatGPT are enabling rampant cheating and plagiarism in schools, as student use these technologies to automatically complete writing, math, and coding assignments.^[6] The AI-detection tools employed by teachers in response have been found to be ineffective and inaccurate – characterizing the entirety of the US Constitution as AI generated, for example.^[7]

Daydreaming. When an AI tool produces a result that is not grounded in reality, it is said to be “hallucinating.” Text-generators such as ChatGPT do not fundamentally understand the text they are producing. They calculate one word or symbol at a time – if they estimate that the next word/symbol they output should be a period, then the sentence ends. Otherwise, the sentence continues.

These systems live in a world of language,” said Melanie Mitchell, an A.I. researcher at the Santa Fe Institute. “That world gives them some clues about what is true and what is not true, but the language they learn from is not grounded in reality. They do not necessarily know if what they are generating is true or false.”^[8]

The permanent record. Just as humans cannot intentionally forget information they have learned, it is not currently possible to remove data from a trained AI tool.^[9] Unlike an Excel spreadsheet, which stores data in neat columns, AI tools store data in the connections between neurons in a deep neural network. Every connection is influenced by every piece of training data, and a large model like ChatGPT-4 is reported to have more than 1.7 trillion connections.^[10] It is not possible to specifically alter these connections in order to remove data without fundamentally changing the model; as a result, for data to be removed, the model must be retrained from scratch.

Teacher substitute? The overall effect that the widespread adoption of AI – especially GenAI – will have on labor is unknown. In 2020, the World Economic Forum published a report suggesting that 97 million new jobs may be created, while 85 million jobs may be displaced.^[11] However, the report goes on to state that though job creation currently outpaces job destruction, the rate of creation is slowing while the rate of destruction continues to accelerate.

AI may bring both benefits and risks to the education workforce. AI may help improve teachers’ practice by completing repetitive tasks like grading, lesson planning, scheduling, and routine paperwork, freeing up their time for direct instruction. Analysis by McKinsey and Company suggests that AI could help teachers reallocate 20% to 40% of their time to activities that support student learning. But research also indicates that teachers are concerned about the impact of AI on their profession.^[12] Some fear that, in the drive to personalize instruction and optimize for efficiency, teachers, and their pedagogical skill and human connection with students, will be devalued and they will ultimately be replaced. As the implications of AI on the teaching profession become clear, teachers will require opportunities to learn how to use AI and how to use it to support, rather than supplant, the unique skills that teachers possess.

1. Author’s statement. According to the author:

Artificial intelligence has the potential to assist teachers and enhance quality of education. It also has the potential to enable plagiarism, reduce critical thinking skills and displace essential workers. AB 2652 directs the Superintendent of Public Instruction to convene a working group comprised of various stakeholders to thoughtfully address how schools can best integrate this new technology so that it works with us, not against us.

1. What this bill would do. This bill would create a working group to study the potential risks and benefits of AI in the education space. The working group would focus on both students and teachers. For students, the working group would emphasize education outcomes, student privacy, and equity. For teachers, the working group would additionally emphasize the effect of AI on the education workforce. The working group would be tasked with developing guidance for local educational agencies and charter schools, as well as developing a model policy for these entities regarding the safe of effective use of AI.
2. U.S. Department of Education. Recognizing the powerful influence AI will likely have in education, numerous organizations have called for the development of policies and guidance around the use of AI in education. The U.S. Department of Education notes that:

Policies are urgently needed to implement the following:

1. Leverage automation to advance learning outcomes while protecting human decision making and judgment;
2. Interrogate the underlying data quality in AI models to ensure fair and unbiased pattern recognition and decision making in educational applications, based on accurate information appropriate to the pedagogical situation;
3. Enable examination of how particular AI technologies, as part of larger edtech or educational systems, may increase or undermine equity for students; and
4. Take steps to safeguard and advance equity, including providing for human checks and balances and limiting any AI systems and tools that undermine equity.^[13]
5. TeachAI. In 2023, TeachAI, in collaboration with Code.org, CoSN, Digital Promise, the European EdTech Alliance, James Larimore, and Policy Analysis for California Education (PACE), launched an AI Guidance for Schools Toolkit to help school systems meet the urgent need for guidance on the safe, effective, and responsible use of artificial intelligence.

The AI Toolkit highlights seven key principles for educators to consider in developing guidance on AI and education for their staff and students:

1. Purpose: Use AI to help all students achieve educational goals.
2. Compliance: Reaffirm adherence to existing policies.
3. Knowledge: Promote AI literacy.
4. Balance: Realize the benefits of AI and address the risks.
5. Integrity: Advance academic integrity.
6. Agency: Maintain human decision-making when using AI.
7. Evaluation: Regularly assess the impacts of AI.^[14]
8. Analysis. Just as the invention of the internet and personal computers fundamentally changed education more than 30 years ago, the widespread adoption of AI in California’s schools promises to affect how students learn and teachers work. These technologies could be transformative: in the long term, AI mentorship and custom lesson plans could lead to every student in California having the equivalent of a personalized tutor at their fingertips. In the short term, beneficial AI technologies are unlikely to be distributed equitably throughout the state. The working group that this bill creates should focus on ensuring that the benefits wrought by AI are not limited to a privileged few.

As the working group proceeds, it should prioritize educational outcomes and student safety. This includes evaluating how AI tools impact learning processes and outcomes, and ensuring these technologies do not compromise student welfare. Furthermore, the group should consider what training will be necessary for both students and faculty to safely and effectively engage with AI technologies. This training should address the capabilities of AI systems, the ethical considerations in their use, and strategies for mitigating risks associated with these technologies.

Even now, the unregulated use of AI is raising significant concerns regarding privacy, security, and ethics. Children and teenagers are known to be susceptible to misinformation and disinformation, phenomena that AI can exacerbate. Notably, there have already been troubling instances where AI has been misused by students, such as the creation of non-consensual deepfake pornography. These incidents underscore the urgent need for robust guidelines and educational programs that empower students to recognize and resist such abuses of technology.

The working group should explore potential solutions to these problems, offering recommendations for educational institutions to better prepare students to confront and counter the negative implications of AI. Through comprehensive study and proactive strategy development, the group can help ensure that AI serves as a beneficial tool in education rather than a disruptive risk. Above all else, the workgroup that this bill creates should be neutral in its approach. AI has the potential to level the playing field with respect to educational resources. Conversely, it could serve to amplify historical inequities. When it comes to student education, new technologies and approaches should not be immediately discounted.

1. Author amendments. In collaboration with committee staff, the author has proposed several amendments aimed at expanding and clarifying the questions that will be asked of this working group. Conceptually, the working group shall:

• Assess the ownership structure of student- and educator-created materials resulting from the use of advanced technologies.
• Assess the role that student and educator consent should play in the use of AI technologies.
• Develop strategies to ensure that student and educator feedback is continuously collected and considered as AI technologies become more widely adopted.
• Assess the impact of AI technologies on employment and labor dynamics within the education sector, including the relationship between job enhancement and replacement.
• Develop strategies to ensure that the adoption of AI does not exacerbate existing inequities throughout the education system.
• Develop guidance related to the adoption of artificial intelligence technologies that augment educators’ ability to teach students.
• Develop strategies to ensure that educators receive adequate training, fair compensation, and opportunities to offer feedback and guidance both individually and as a collective.

1. Related legislation. AB 2370 (Cervantes, 2023) would prohibit the replacement of faculty by AI, and would restrict the use of AI to “peripheral tools” that provide support to faculty. This bill is pending before this committee.

ARGUMENTS IN SUPPORT:

PERK Advocacy writes:

PERK supports AB 2652 because artificial intelligence (A.I.) is a modern technology with exponential growth and with that the potential to harm a student’s privacy. This bill would require the Superintendent of Public Instruction, to convene a working group for the purpose of exploring how A.I. and other forms of advanced technology are currently being used in education and developing best practices to ensure that those technologies advance, rather than harm, educational quality. PERK believes we need to better protect pupils and teachers from the potentially harmful effects of A.I. and other forms of similarly advanced technology now in classroom use across the state.

The California Teachers Association writes:

AI at its best holds the promise of allowing certificated and classified workers more time to focus on interacting with students and less time on lower priority tasks. For example, versions of this technology collect data inputs from teachers and organize it such that much less time is wasted collating documents or struggling with spreadsheets. More advanced AI can help teachers prepare lesson plans, develop appropriate and effective tests, and help personalize strategies to make the learning process more engaging for students.

However, the risks posed by AI in the education process cannot be overstated. Generative AI, for example, allows anyone to input parameters and quickly generate text, images, videos, or other outputs as if the individual had produced the work themselves. Other forms of AI allow anyone to highlight any text and use software to prepare a slide deck on the topic, generate a quiz based on the text, and even offer technical feedback on related essays written by students. While the time-saving potential of such technology is significant, so is the risk that it could someday be used to replace critically important components of the learning process while devaluing, deskilling, or even displacing teachers and classified workers.

Los Angeles County Office of Education writes:

This bill takes a proactive and forward-thinking approach to addressing the evolving role of AI and advanced technologies in education. By fostering collaboration and research among stakeholders, this bill will enable informed decision-making and promote the responsible and effective use of these technologies to benefit students and educators.

REGISTERED SUPPORT / OPPOSITION:

Support

California Federation of Teachers Afl-cio (sponsor)

California Association of School Business Officials (CASBO)
California School Employees Association
California Teachers Association
Los Angeles County Office of Education
PERK Advocacy

Opposition

None on file.

Analysis Prepared by: Slater Sharp / P. & C.P. / (916) 319-2200

1. Netflix, "How Netflix’s Recommendations System Works," help.netflix.com/en/node/100639, accessed on Feb. 22, 2024. ↑

2. OpenAI, "How ChatGPT and Our Language Models Are Developed," help.openai.com/en/articles/7842364-how-chatgpt-and-our-language-models-are-developed, accessed on Feb. 22, 2024. ↑

3. Rohit Sehgal, "AI Needs Data More Than Data Needs AI," Forbes, Oct. 5, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/10/05/ai-needs-data-more-than-data-needs-ai/. ↑

4. Aditya Malik, "AI Bias In Recruitment: Ethical Implications And Transparency," Forbes, Sep. 25, 2023, https://www.forbes.com/sites/forbestechcouncil/2023/09/25/ai-bias-in-recruitment-ethical-implications-and-transparency/. ↑

5. Claudia Lopez Lloreda, "Speech Recognition Tech Is Yet Another Example of Bias," Jul. 5, 2020, www.scientificamerican.com/article/speech-recognition-tech-is-yet-another-example-of-bias/. ↑

6. Kalley Huang, "Alarmed by A.I. Chatbots, Universities Start Revamping How They Teach," New York Times, Jan. 16, 2023, www.nytimes.com/2023/01/16/technology/chatgpt-artificial-intelligence-universities.html. ↑

7. Benj Edwards, "Why AI detectors think the US Constitution was written by AI", arstechnica, Jul. 14, 2023, arstechnica.com/information-technology/2023/07/why-ai-detectors-think-the-us-constitution-was-written-by-ai/. ↑

8. Cade Metz, "What Makes A.I. Chatbots Go Wrong?," New York Times, Mar. 29, 2023, www.nytimes.com/2023/03/29/technology/ai-chatbots-hallucinations.html. ↑

9. Stephen Pastis, "A.I.’s un-learning problem: Researchers say it’s virtually impossible to make an A.I. model ‘forget’ the things it learns from private user data," Yahoo! Finance, Aug. 30, 2023, finance.yahoo.com/news/un-learning-problem-researchers-virtually-164342971.html. ↑

10. Reed Albergotti, "Microsoft pushes the boundaries of small AI models with big breakthrough," SEMAFOR, Nov. 1, 2023, www.semafor.com/article/11/01/2023/microsoft-pushes-the-boundaries-of-small-ai-models. ↑

11. World Economic Forum. "The Future of Jobs Report 2020," Oct. 2020. ↑

12. Nikolas McGehee, “Balancing the Risks and Rewards of AI Integration for Michigan Teachers,” Michigan Virtual, Dec. 5, 2023, https://michiganvirtual.org/research/publications/balancing-the-risks-and-rewards-of-ai-integration-for-michigan-teachers/. ↑

13. Office of Education Technology, “Artificial Intelligence and the Future of Teaching and Learning,” May 2023. ↑

14. Hadi Partovi and Pat Yongpradit, “AI and education: Kids need AI guidance in school. But who guides the schools?” World Economic Forum, Jan. 18, 2024, https://www.weforum.org/agenda/2024/01/ai-guidance-school-responsible-use-in-education/ ↑