Senate Select Committee on Select Committee on Career Technology and the New Economy

Good morning. Welcome to this Select Committee hearing of the Career, Technology and New Economy Select Committee. We're convened this morning to talk about Senate Bill 755 and the findings of the report that the Administration prepared and released on March 13,2024. We're talking about measurement. We're talking about data points.

We're talking about how we improve data collection and put the data together to improve what we do in terms of our workforce training programs and the connectivity between those programs and jobs that actually exist. You know, I've been thinking about this issue for a long time. When I first came to this California State Senate about 12 years ago, for some reason, I was taking a look at one of the local workforce development board publications that happened to be made available to me.

And I was looking at it to review statistics on training programs and job placements, specifically to see if we were actually training people for the jobs that actually exist. And if people trained, were they securing jobs in the fields and occupations for which they had been trained? Of course, you know what I found in this particular case? I didn't find much of anything in that particular report, and I wondered why.

I wondered if it was specific to the workforce development Board report in question that I was looking at. But if not, I wondered how as a state, we were actually able to manage the money that flows, flowed and continues to flow through the state, federal and state money, in some cases maybe well over $1.0 billion or more every year. If we don't have the data necessary to actually eliminate duplication in programs, achieve efficiency, and deal with those training programs that simply do not work.

So I thought, well, I'd like to do something about that. It was a little difficult to get traction at first. I was sort of busy. My team was sort of busy, other people were busy. And then we had the pandemic, which, of course, emphasized, I think, to all of you and certainly to us, the importance of having more and better data about our workforce. You know, the benefits seem pretty clear to me.

You know, more information on the workforce by occupation instead of the general industry related data that we have, seem to have today, which doesn't allow us to dig down and find out what's happening in particular occupations data to analyze the question, pretty central question, is our workforce today full time or part time, and our workers employed in career track positions where they can actually have upward mobility and secure hire and, and better wages, and, of course, data to address the issue that started me down this path 12 years ago.

How effective are our training programs and our training providers in delivering jobs and skills training that are aligned with the intended occupations. You know, do jobs actually exist in sufficient numbers for those people that we are actually training? And I don't know that we know the answer to that. Maybe we'll learn today from those of you who are going to come up and talk to us.

But as I said, traction was difficult in terms of trying to get a handle on this and try to put a fix forward. But fortunately, in 2022, with the help of a very, very, very talented lege consultant, Dan Rounds, to my left, and my staff, specifically including my Chief of Staff, Gilbert Martinez, and my principal consultant, Kearney King, to my right.

The Legislature we put forward, and the Legislature actually passed Senate Bill 755, a Bill which directed the EDD and the California Workforce Development Department Board, rather, to develop a plan to implement new program performance and program evaluation methods for job training services, specifically establishing three new outcome measures related to how individuals who receive job training training do or do not succeed in the labor market.

Training related job placement, job placement at or above a regional living wage, and upward mobility over time through the Employment Development Department's base wage file, new data points will need to be collected about workers and their employers, specifically occupational code data, hours, work data, location of employment, as the report illustrates. And we're going to go over that some of this. Clearly, there are going to be challenges in implementing SB 755 and the mandate that is contemplated in that Bill. There are going to be system challenges.

There are going to be budget challenges. There are going to be challenges to the employer community in a variety of ways, both in terms of systems education information and the like. However, in looking at the report, frankly, none of these seem insurmountable to me, whether it's the budget issue or the training requirements or the system upgrades. And it seems to me as well that the benefit to the State of California certainly outweighs the effort it will take to get there. So that's what we're about today.

We have two panels. The first panel includes representatives from the California labor and Workforce Development Agency, the California Employment Development Department, and the California Workforce Development Board. This panel, number one, will discuss the administration's report, their current workforce programs, how they use the data they are currently collecting, and how improved and augmented data collection as proposed by Senate Bill 755, will facilitate additional analysis of workforce program outcomes. Program evaluation for individuals who receive job training I think I've said it, but I'll say it again.

Given California's current budget deficit, it seems to me that it's imperative we focus on our workforce development investments and that we make sure that those investments are directed toward programs that work. Our second panel consists of scholars and researchers from UCLA, UC Berkeley, and the Public Policy Institute of California. And the panel will provide testimony on the value of collecting and analyzing data using the base wage file. That's enough of me talking.

Why don't I invite our first panel up in the well to the table to start the presentation? Thank you for joining us. And I'm going to ask you, once you're seated, to introduce yourselves. Starting to my left, your right. There is at least one name in here that I'm sure that I will not get right. And I do not like to mispronounce names. So why don't we start to. Okay. Yes, ma'am.

Good morning. Jennifer Sturdy with Deputy Secretary of Evaluation with the Labor and Workforce Development Agency.

Thank you, Miss Sturdy. Is it working? I can't. There we go.

Good morning, Senator Roth, my name is Javier Romero. I'm the Deputy Director of the EDD Workforce Services branch.

Director Romero, thank you for joining us, sir. Is yours working?

Yeah.

Good morning. Padmakumar “PK” Govindankutty, information technology manager, Workforce Development Board. And you are? PK. That's a short form.

Yeah. Thank you. That will help. Thank you for joining us. Yes, sir.

Good morning. Randy Ginise with the Employment Development Department, Tax Branch, Tax Support Division.

I'm going to kick us off. Good morning, chair. Thank you for inviting us to present to the Committee today. We have three objectives for this panel. The first is to provide a comprehensive overview of current workforce program performance, reporting and evaluation efforts for the programs defined in SB 755. I'll invite Javier Romero to present for EDD's WIOA title one programs, and Padmar Kumar Govindunkuti will present the overview of CWDB's programs and the Cross Department performance efforts under the Cal skills program.

The second objective is to go over a review of what data collection and outcome measures would be augmented if the SB 755 plan were implemented in full. And I'll then summarize the SB 755 plan that was submitted to the Legislature earlier this spring.

Okay. Thank you. Please proceed.

I'll begin with the overview. EDD administers the federally funded WIOA title one core programs, which consists of adult, dislocated worker, and youth.

So let me have you move that microphone a little. Same for all of you. Move. Just the whole thing pulls. There we go. Very good side of that. Or they have to turn up the volume for people like me. Very good. Much better. Much better.

Very good. The funds for these programs are primarily made up of federal formula allocations to the 45 local workforce development areas. Additionally, the 15% governor's discretionary funds allocated to statewide workforce activities can also support this work. While federally funded, these programs operate using program years, or pys, which run from July 1 through June 30. From 2019 through 2023, nearly 1.6 billion was allocated for employment and training services through title I programs.

In the most recent complete program year, program year 2022, which ended on June 30, 2023 the title one program expended 277 million on employment services, which consist of, like, job search, placement assistance, career planning, workforce preparation assistance, and more than 60 million on training services. Next slide.

Okay, they're not up on the screen.

Okay, very good.

But we have them.

Very good. I'll continue my work.

I don't know if they can. They're not up on the screen. Did you intend for them to be up on our screen? Who does that? Might be helpful for people who are watching. So while we're waiting for them to figure this out, the 1.0 it's at 1.57 billion. At least it's here. So about 1.6 billion. That's primarily federal money?

Yes, that's correct.

How much of it was state money? General Fund.

That amount is federal.

Zero, so is there state money on top of the 1.0 Fund?

Yes, but I don't know what that is in front. We could get back to you on that.

Okay. Plus state money, TBD, to be determined. Yes.

Okay. Very good. All right. Very good. EDD uses an it system known as caljobs as the system of record for title I core programs and is required to report participant data, including outcomes, to the United States Department of Labor Employment Training Agency, on a quarterly and and annual basis. DOL has specific guidelines for reporting, including timelines, cohorts to include in the report.

The WUA annual narrative Report is posted publicly by the California Workforce Development Board each year after the report deadline of December 1, and the DOL publishes data for public consumption on their performance results page. In addition to participant reporting, EDD reports the eligible training provider list known as ETPL data, including outcome data to DOL on an annual basis via the eligible training provider report.

This report includes all students, WIOA and non WIOA, served by each of the training programs listed on the California ETPL. The information contained in this report is used annually by local areas and the EDD when determining if a training program can remain on the ETPL, ensuring program funds are spent on quality programs next slide title I Core programs offer employment and training services customized to the program participants needs for program year.

Title one core of title one core program data shows that 64% of the program participants receive only employment services and on average, 36% receive training services, which is roughly three percentage points higher than the programs nationally. And those that went on to training have a higher inter employment rate and median earnings in the second quarter after exiting the program.

Next slide the two evaluations that were recently conducted by the California Workforce Development Board on title one data, Cal skills 1.0 evaluation and the evaluation of the Americas Job Centers of California. Both evaluations concluded that training impacts on employment and earnings are positive and statistically significant. This is in alignment with the PY 2022 outcome data shown on the previous slide. Additional information on both evaluations can be viewed on the California Workforce Development Board's website.

May I ask a couple questions about the bar graph slide that was right before this one?

Yes.

You know, I'm not familiar with this, so if my question is a dumb one, I'm sure you'll feel free to tell me. My scan's pretty thick after 12 years here, but it looks like they're more the total participants receiving job training services are more than the total participants. Am I reading this chart right? The orange line is longer than the blue line. Did I. Am I missing something? I don't know how you're.

oh, let me. Yes, 1 second, please.

So maybe you had participants receiving training two different types of training, maybe.

Yeah. What they were comparing was the outcomes from the total participants and then the total participants that received job training, demonstrating that those that received training, their outcomes were higher. So that's why it seems like, zero, employment rate. Yes, yes, yes. So, okay. Yeah. So basically, you know, it's demonstrating how those that got training exceeded outcomes exceeded those that only receive employment services.

And to the point of Senate Bill 755. I guess the problem with this data is we don't know what kind of employment.

Yes. For the most part.

So we could be training. We could be training a dental assistant who then goes to work in a Starbucks.

Well, we do have data as far as where people get trained and so on.

Well, I'm jumping into somebody else's sandbox. Yeah, but you can just defer the question and to somebody else to answer. But my point is, I mean, we know what they're trained in and we report us somebody somewhere in here. I see. We report certificates or credentials obtained, so we know someone finished a dental hygiene training program.

But our data. Correct me. Our data is not sufficient for us to be able to tell whether that person got a job in a dentist's office as a dental hygienist. Right?

That's correct.

What we do know is they got a. They received a credential after training and they have income.

That's right.

And the income is reported.

That's right. That's right.

That's the disconnect between the efficacy of the training program in terms of job placement.

Right, right. We don't know if, in fact, they're working in the field they were trained in. We don't know the amount of hours they're working in that given time period and so on. That is the disconnect currently.

I think the Federal Government would want that, too, given the amount of money that they shove in the pipeline down to us. Right?

Yes. And I would say that we're hopeful there's ... nationally, there's increased interest in this information. So we're hoping eventually the Feds will consider funding such. Such a thing. We like that. Well, sure. Should that occur.

I derailed partially your train, so you can fire it back up again next. Are you finished?

I believe I'm done. I'm. At this juncture, I will turn over to my colleague PK from the California workforce right now. I'm here.

Thank you, Javier. Good morning, chair. Morning, Members. Thanks for the opportunity. I'll start with the CWDB program overview. Workforce Development Board administers several grant programs that Fund local governments and nonprofits to help historically marginalized populations train for and obtain family sustaining careers. These grants come from a combination of state and federal dollars and are awarded competitively. CWDB currently manages about 350 million in grants, which are expected to serve about 47,000 Californians. Of a 62% will receive job training.

All the remaining funds are encompassed for different CWDB programs. Next slide please when CWDB moved into managing programs, the only source of data capture was Cal jobs. As a number of programs and participants expanded, there was a need for creating its own data capture system with discussions about a long term, stable and secure data capture system.

CWDB created an interim solution called SGDS State grant Data Solutions and further designed a robust salesforce based system called Cal E Grants, which was rolled out over the last six months. Each grant program has its own statutory requirements for performance reporting and also the wage matching requires a one year lag time. Next slide, please this slide shows the performance data for the program year 20212022 across three CWDB programs, p two e person to employment, high road construction carriers, and high road training partnership HRTP.

Collectively, about 62% of the participants served, received job training. Notably, a significant number of p two e participants had already earned industry recognized credentials from job training they received while incarcerated. Program funding is also focused on providing services to support individuals re entering the workforce. In most cases, participants who received job training far better in the labor market. For example, the second quarter employment rate among HRTP participants was 70.2 percentage, and that number was about five percentage points higher among those who completed training.

Furthermore, the second quarter median wage among HRTP participants was 13,761 and that number was about $1,500 higher among those who completed training. Next slide, please. CWDB's independent evaluation strategy CWDB is focused on continuously improving California's workforce system and third party evaluations of our grant programs are key to this effort. These evaluations report typically examine best practices for serving historically marginalized populations. Recent p two e valuations have helped uncover what exactly works.

That is, a training that leads to credential, early reintegration into the labor market, and supportive services, which includes emotional and psychological support. Next slide, please. Workforce Development Board designed a robust data platform called Cross Systems analytics and assessments for learning and skills attainment, in short, Cal skills. It was established to satisfy UIC section 140, which requires CWDB to develop a workforce metrics dashboard that measures investments in workforce development to better understand their impact on the labor market.

Calskills is a multi departmental and interagency effort that integrates, evaluates, and reports data from 13 workforce development programs across eight data sharing partners. Cal skills reporting includes descriptive statistics and summary analysis via workforce metrics dashboard report as well as statistically rigorous evaluation of program impact on participant outcomes via CPL report, which is a California policy lab. Who is our academic evaluation partner? Next slide, please.

Calskills uses common performance measures to examine participant characteristics, participant outcome, and other workforce system factors, including services provided, provider location and industry of employment. Metrics are currently informed by Department of Labor reporting requirements and UI code 140, which includes total number of participants served, exited and completed turning for a fiscal year and also percentage employed second and fourth quarter after exit, median quarterly earnings second and fourth quarters after exit and the percentage attained prudential fourth quarter after exit. I'll stop here. Thanks for listening.

I'll pass it to my colleague Jennifer Sturdy to go over the SB 755 details.

Thank you. Let me ask a couple questions on your first overview slide. Funding from this is PY 2019 to 20 to 2324 I guess $540.8 million. It says does not include pass throughs. What does that mean? I was looking at this slide. This is the overview slide. 540.8 million funding for those years does not include pass throughs. What's the pass through? Is that federal pass throughs?

Sorry, if I could step in. Those are state funded programs that go through the California Workforce Development Board. They're not issued through competitive award process or solicitations.

Are there earmarks? But the earmarks out of the General Fund for Workforce Development, this is program funding. So somebody just gave somebody $540.8 million, right?

Yes.

And then they're pass throughs. Are the pass throughs over and above the $540.8 million?

Yes.

And those are out of the state General Fund.

I think a variety of sources of funding.

How much does that entail?

The main share of this Fund is coming from the Veola 15 percentage discretionary Fund and the pass through, which you mentioned. I can get you the breakdown of what exactly that is.

Right. Because what I'm trying to get a handle on is how much money we spend on workforce development, training and other things, but primarily training. And the reason it's difficult, I guess, for me to get a handle on it is so many buckets coming from so many different directions. And I think it's important to know what the total is because the reporting that we're talking about impacts or involves the total.

So when you're talking about a budgetary expenditure to get the systems up and running of 25 million or 50 million or whatever, it turns out to be, you have to measure that against the amount of money rolling through the system over time. Right.

Understood. Yes.

So it would be helpful to know what that extra money, what it totals and where it comes from. I suppose we could look at the budget, but the budget, you know how many volumes that is? And we'd have to figure out where the line items are and what they are. And you all probably have a better way to compute that than me. 1.6 thousand active current awards. Are those. Is that the number of participants in the CWDB program?

That's currently being recorded in the CWDB program.

So those are the number of participants. We didn't get that from the WIOA slide. I don't think I missed it. Maybe somebody can go over that when we go back. I didn't see the number of participants in the various WIOA programs. That was one of the questions that we asked. Taking a look at your slides. Somebody's going to have to educate me after this hearing as to what the second quarter employment rate.

So does this slide reflect the fact that some participants only seek services for employment but not training and employment? Is that why the orange is 55.2% and the blue is 58.1%?

Yeah, that's specifically for the p2e program, which our researchers also looked at it. Because the participant got trained. The amount is less. That's because the training that the participant went through while they were incarcerated was that that training got them into the labor market and they got employed. But those participants who got training right after, when they came out, they were delayed in getting. So that is the discrepancy that you see there.

I see. Okay. Thank you. Thanks for that clarification. Now the cow skills system, refresh me. Is that up and running?

It is. We are in the process of re engineering the complete calskills. The Calskills, as is the legacy Cal skill system exists now we are migrating that entire system into cloud. Aws. Go cloud.

It's phase two, right?

It's a phase two. Correct.

And then we have a phase three.

Phase three will have. I mean, we are anticipating, we are hoping that SB 755 will be a part of that. And our first set of Cal skills report came out from our prototype calskills, version one. And the second version is what? Currently we are migrating to the cloud and we have re engineered the database in such a way that it can accommodate any kind of variable data that comes from different partners. Currently we have 13 programs sending us data and it's all in non uniform format with the new system.

We came up with the standard uniform layout in such a way that they can provide the data in that standard layout and it can get into the system without any much manual work.

Now, I noticed in the six items that you have on your slide in terms of the reporting requirements and metrics, we have median quarterly earnings second and fourth quarter after exit and percent attained credential four quarters after exit. Is this Calskills system able to do?

One of the things that we hope to accomplish with Senate Bill 750 and the implementation is to get some idea of career progression, not only regional living wages, but career progression over a career, getting data at certain phase points more than just the first quarter after the training?

Yes.

Is your Cal skill system able to do that?

Not with the current set of data that we are receiving. With the additional data that we may receive with the implementation of SB 755. Yes, that is possible.

I see. But the system itself can generate median quarterly earnings over a period of time for particular groups of people.

Yes.

And one other question that goes to sort of the relationship between CalSkills, the Workforce Development Board, and the WIOA reporting. How does all that relate? I mean, does Cal skills do it all for the WIOA programs as well? Is that a fair question? Am I going to get a couple of different reports, whether it's workforce development or WIOA?

Yeah, there are two reports that comes out of the Cal skills. One is a workforce development workforce metrics dashboard report. And the second one is the statistically rigorous evaluation done by California Policy Lab.

So the dashboard report will have all these measures that is being listed here, and the analysis report that comes from California Policy lab will have more detailed, descriptive information about which program did better for a given program year and what are the measures that we need to take in order to go to the next level.

Let me ask you another way, because you can tell that I'm not, I didn't never Chair the Labor Committee here, so I don't get involved typically in these issues, except this one interests me. So we have 1.57 billion over a period of years, and we have 500 something million at the board level over the same, apparently the same period of years.

Is this CalSkills when it's fully implemented, whether it's phase two, phase three, whatever, will it take all the programs funded by $2 billion and deliver data, or is it going to be chimneyed piped, as we say?

Calskills will accommodate all the existing programs that are currently there in Calskills, plus all the workforce development board programs that are not currently available in Calskills system. So that's a plan. So 13 programs that currently Calskills accommodate, plus all the CWDB programs.

Are there workforce development programs outside of what you just described that will not be captured by Cal skills?

I'm not aware about that. I can find out.

Is that an intelligent question? You can tell me? No, I'm just trying to figure out, are we going to have one system that produces a report so someone like me can look at PowerPoint slides like this and understand where we're spending our money and whether the data points that we referenced in SB 755 or even the data points that are currently listed under Cal skills, what the data is.

You're asking the right question. I mean, Calskills, as a current system, has a current scope of workforce development programs that are within it.

That's what I thought.

And that scope currently does not include the California Workforce Development Board programs that they have been implementing. And so that's why in SB 755 plan, it's part of the additional resource needs to expand the scope of CalSkills to include those additional programs.

And that's part of your presentation. And I can certainly wait for that. But you know, my goal, not that I get to have one, but I'm going to have one, my goal is that all of the workforce development programs will feed data, whatever the data points ultimately wind up being into a system, an it system that is able to tell us and answer the questions that I think need to be asked. You know, who's, you know, how many are being trained, what the training is.

Does the training match up with actual jobs that people get at the end of the pipeline? What wages are they earning? Are they what we would call living wages for regions in the state to be determined.

And then ultimately, the training leading to jobs, to the jobs provide the type of career path that we want for people in the State of California, which would show or would demonstrate in the form of incremental increases in wages over time, regardless of the source of the funding and regardless of which agency, Department, or board is administering the funding and the grants and all of that. That's what I hope to see. And I guess you're going to be talking about that, right, ma'am?

Hopefully, yes.

I think that's all the questions I have right now.

Thank you.

Welcome, Senator Ochoa Bogh. Okay, who's next?

Thanks, PK. I'll take us to the second objective of the panel. So now I'll provide a review of what data collection and outcome measures would be augmented if the SB 755 plan were implemented in full. A reminder that SB 7505's intention is to build on the existing data infrastructure to collect more data related to job training participant outcomes for WIOA title one programs managed by EDD and CWDB programs, as well as to identify the necessary resources to add that data collection.

So far, we've talked about existing data infrastructure for workforce participant data systems such as EDD's caljobs. In addition, you've already raised the base wage file, which is managed by EDD and is another existing data infrastructure that analysts rely on for actual employment and earnings over time. This data is submitted by all employers in the state and is primarily used to calculate Unemployment Insurance and other benefits. So what data collection analysis would be needed if outcome measures were expanded? We have four different potential expansions.

The first one is training related employment. To do training related employment, we need the occupation of the training which we have in our participant data systems. We would also need the occupation of employment which we do not have in the base wage file. The second element, we are asked to compare participant earnings against a specific regional living wage definition. To do this, we would require defining what is a regional living wage.

We would also require median quarterly earnings, which we currently can obtain over time from the base wage file. However, we would also need hours worked, the region where the individual is working, and the region where the individual lives in order to fully calculate the living wage. Those elements are not currently in the base wage file. Third, we are asked to track upward mobility of job training participants over time by expanding.

When we measure their employment and earnings according to SB 755, this would be up to one year before training takes place and up to three years after training. This requires expanding how long and how often analysts merge participant data to the base wage file to track on employment and earnings over time. This is feasible to do with our existing data system and in fact was already done in the Calskills 1.0 evaluation analysis that was mentioned earlier.

Finally, we are asked to compare the differences in outcomes for job training participants by key characteristics, such as the training provider who delivered the service, the region the job training took place, as well as the industry and occupation of the training. All these data points are in our existing participant data system, so we can do this analysis. The biggest constraint to this analysis is based on programs scope and the actual sample sizes within the data. For example, it can be challenging to do regional analysis of a specific program only operated in one or two regions.

What's occupation of training mean? I see that that's at the top and at the bottom of this particular slide, does that mean we're training somebody to be. I'll use my dental hygienist example exactly so. But the connection on this last one is not necessarily occupation and training, but occupation and employment. Right.

For the last one, it is the.

We want to find out if somebody actually gets a job as a hygienist, because, of course, it's been a little bit of time since I've seen an article in the newspaper, but there were these anecdotal stories about we're training, you know, 250 coders in San Diego, and there are 10 jobs available for the 250 we've just spent training. I know that was an article. I don't. My numbers are probably a little off, but not by much.

Seems to me as a state, we want to try to avoid that because that's a lot of money. I won't say wasted, but we spend a lot of money training people for jobs that apparently don't exist there. Maybe they exist somewhere. So we need to move. Occupation and training data is important, but we also need occupation and employment data.

Yes, and that is the first data element that was mentioned. So making sure that we can measure training related employment placement.

And wouldn't the sample size, wouldn't it be large enough to determine first?

So for some programs, it depends on the total number of participants, where it operated, how many industries or occupations individuals were trained on. Whenever you want to cut the data more and more ways, it gets smaller and smaller. So just flagging that any sort of disaggregation of analysis by that many other elements can start facing sample size issues.

Okay, I didn't mean to totally interrupt, but I did. Thank you, though.

Okay. With that, I'll move to the last objective of the panel to provide a summary of the SB 755 plan that was submitted. The labor agency, EDD, and CWDB staff worked collaboratively since January 2023 to deliver on the SBA plan. First, we want to flag there are existing work streams and resources that support EDD and CWDBS workforce performance analysis and evaluation efforts that are relevant to SB 755.

For example, in 2023, labor agency and the SB 755 working group led and completed several relevant work streams, such as establishing a centralized workforce program inventory. Exactly.

To your point, to kind of know the scope of all of our programs also defined a set of workforce common metrics relying on input from the SB 755 expanded outcomes, as well as existing performance metrics that are in UI code 14013, as well as reviewing all the existing participant data systems for alignment with the data elements that are required to report on those metrics.

Next slide in 2023, CWDB and the Working group led the assessment of the existing Calskills program and its data infrastructure system for relevance to SB 755. As presented in the plan, CWDB is currently modernizing the Cal skills system to move to a secure, cloud based environment, improve data intake and standardization, and produce an interactive, public facing dashboard, all of which align with the original performance analysis elements that are defined in UI code 14013.

So moving toward a stronger system, but still one that's built around the original performance metrics that guided Cal skills. In addition, in 2023, CWDB co created the second phase of the Cal skills evaluation analysis with their academic research partner, the California Policy Lab.

The scope of the workforce programs continues to include that original scope from Cal skills, which PK listed earlier before the second phase evaluation analysis scope of work references expanded outcomes such as upward mobility and disaggregation elements such as training provider, region, industry and occupation, and requests. These are included in analysis as feasible based on sample sizes and data availability.

The second phase evaluation effort is already funded through WIOA 15%, and the contracting process has begun with an expected period of performance from summer 2024 to summer 2027. Finally, in accordance with the Bill, the working group identified the activities, costs, and timelines that would be required to adapt these existing systems to align with the expanded requirements of SB 755.

As noted at the bottom of the slide, implementing the SB 75 plan as written would require 25.4 million in one time costs and would require between 5.6 and 9.4 million in annual costs to maintain the changes. The largest costs are attributed to the expansion of the base wage file we've mentioned earlier so that it can include the additional data elements that we have flagged, hours worked, occupation, region worked, and region live. Edd led the assessment of the base wage file system to reach these numbers.

The SB 755 plan documents the significant it and process related adjustments that would be required to the data intake forms, data intake take, infrastructure, and the employer related support and training that EDD delivers. EDD also led a review of lessons learned from other states that have considered or already implemented expansions to the base wage file.

Lessons learned include undertaking the burden, understanding the burden placed on employers to adapt their payroll systems to revise data intake requirements, providing technical support to employers on understanding occupation related data entry, and incorporating data quality assessment and review into data entry workflow. These were noted very important from other states. EDD built the time and financial estimates around a no penalties approach, meaning there would be no penalties to employers who are unable to provide any of these specific new data elements in the SB 755 plan.

EDD's assessment outlines a phased approach to expanding the system and phasing in large and then small employers, with completion of the expansion possible within five years of appropriation and legislation. The remaining set of activities relates to the expansion of the Calskills system. CWDB estimates 500,000 in one time cost to expand the set second phase of the Cal skills evaluation analysis so that it can include CWDB's programs.

The evaluation analysis will remain limited regarding training related employment and regional living wage analysis unless the base wage file is enhanced. CWDB estimates it would require 500,000 annually for staff time to expand the Calskills data intake and dashboard system in order to add the CWDB programs and align with the expanded analysis elements that are defined in SB 755.

We also estimate $300,000 in one time cost to independently assess the Cal skills system roles and responsibilities to assess its current placement within CWDB given the potentially expanded scope and evolving role of CWDB in job training program design and implementation. And finally, for the expanded analysis and reporting requirements, EDD would require an estimated 150,000 annually for staff time to conduct expanded base wage file merging to participant data systems, and EDD and CWDB estimate 85 to 255,000 for additional staff time for the expanded reporting requirements. That brings me to the end.

I have to say that the, the cost estimates for implementation seem incredibly reasonable for the State of California, having spent a lot of time on budget sub 4, 6 years, actually chairing that when we had all of our other it programs moving ahead in California, and 25.4 million probably was some cost of some study along the way for one of those. Let me ask a question.

So how do you evaluate those of you at the board or in the agency, how do you evaluate overall the training programs that are in place right now in the state? In other words, quality control. You have data points that show numbers of participants, participants credentials awarded, and you have income, presumably from a job of some kind.

So when you look, do you have a program, where do you have a process where you evaluate from a quality standpoint, not just quantity, but from a quality standpoint, training programs that you, we have funded? Is that a fair question?

I can jump up with one answer. So there are a few ways in which we conduct performance analysis and evaluation. So, for example, standard WIOA performance analysis reporting meets federal requirements for CWDB, there is an independent evaluation process. So many of the programs they may not have a standardized set of common metrics and performance analysis reporting requirements, but most of the CWDB programs have an independent evaluation that is designed and implemented alongside it and then eventually publicly published.

Well, notwithstanding what data elements we currently collect or do not collect. So if I operate a training program in Southern California and I'm going to train, I don't know, welders, and I get a. I'm funded by the agency or by the Workforce Development Board to train welders. And I at the end of each year, I turn out 100 individuals with a welder certification. That's reported, I assume.

And then, of course, you do the data collection on the data points you do collect, and you find out that these welders have an average annual income of $15,000 per year. We don't know whether it's doing welding or whether it's a barista and Starbucks or McDonald's, or doing something else, maybe something else in construction, I gather. If that's what I told you I was going to do, then I get a gold star and I'm okay to move forward, right?

Yeah, I would say under the current system, I move forward. I continue, I submit my application for funding for the next year, and I'm funded because I meet the performance metrics that are in existence today. Right.

When we gather the data, which includes a program start date and program end date, when the participants got and what kind of training they went through and when did they exit, and also the demographics, information about the whole participants, plus the base wage information, we will have a clear idea.

When we do the analysis, we have a clear idea of where did he start during the program in our training process, how much was his earnings and right after, when he's done with the training, how much, you know, he started earning right after the program or right after, like second quarter exit of the program. So we do that comparison and we'll be able to figure out whether the program really worked and the training really gave him the proper outcome for the participants.

But is that really true? Does your analysis take into account increases in the minimum wage, for example, or market conditions in particular regions which drive wages up?

Yes. I mean, that is a missing piece here, and which SB 755 is going to add up. We are going to add, I would say, any additional information, any additional data would make it mean meaningful for our aggregates and other reporting, which we are providing. The addition of occupation of training, occupation of employment, that alone goes a long way.

And also the measurable skill gain rate and how was worked prior to program entry hours worked, post program exit and living, current living wage and the cost of living for the area, the regional planning unit where they are located, all those can get, you know, we can create more meaningful report with all these additional information that we are going to collect.

I agree, PK. I was just trying to, I mean, it seems to me we're not, we don't know whether we're taking credit for things that are outside of the control of the trainer and the training program or not. And the question is, you know, is that the proper place for our training dollars or should we shift them someplace else?

If you only need 10 welders and you're funding 100 slots in a training program every year, and you only can fill 10 of them in a particular region, then maybe we're not directing our training dollars to the right training program. Making sense.

Makes sense.

So when you do your analysis of training programs, do you have a process by which training programs are canceled because they're not productive?

Currently, we don't do the comparison of the program by looking at which program works better on certain geographical area. That comparison is not done in Calskills.

So we really need to work on our labor, our labor outcome analysis here and whether we're really improving people in terms of their jobs and careers. Right? Yes. Well.

Well, is there a. Do we insist, for example, that training programs that receive money, regardless of which bucket it comes out of, provide participants. This is what we're trying to do in SB 755, provide program participants with performance data on how they can expect to perform once they complete the training programs. Do we do that today in some other way?

We do not, no. And that's why it's included as an additional cost that's in the SB 755 plan for additional reporting.

Well, then, so the cost of implementing 755, if it turns out to be this costs increase all the time, let's say $31 million first year and then ongoing six to 10 million year after year after year.

Did you project any sort of savings in the kind of analysis of training programs that we're contemplating with SB 755, where we eliminate the inefficient and ineffective ones and shift money either back into the state's coffers or back to the Federal Government or into other training programs that are more effective? Can we project.

It's a good question, but that sort of cost benefit analysis wasn't a part of the SB 755 plan. But I think it's a good question.

Are we able to do it with the data we have currently?

I mean, essentially, you need an accounting of full costs and a way of accounting for full potential benefits for what you're able to achieve. And I'm not sure yet how we would do that, but it certainly is an interesting question.

Well, if you have training programs that are training people for jobs that either don't exist or they're training more people for jobs than currently exist, that has a cost element to it, doesn't it? In terms of the training dollars we're putting into the training program.

It does. I mean, one thing I would mention with especially the sort of evaluation analysis that we're looking at here, especially the sort of rigorous causal impact analysis that was done by California policy lab for looking at programs like WIOA, title I. There's also a timing issue sometime, and the timing of when results come out, especially when you want to look at three years post training, you know, when those results come out and when we're making funding decisions sometimes have different cycles.

So it's just always important to remember kind of the different timelines of when the results of rigorous evaluation analysis, rigorous performance analysis are available and when funding decisions are made for different budget cycles.

Now, in terms of some of the decision points that you referenced in the report were implementation, the no penalty versus penalty issue, which I thought was a significant one. But the system currently is a no penalty system, right? In terms of data collection.

So currently the system is no penalty.

And is it working?

Absolutely.

Compliance rates.

People comply with the data that they're supposed to report. Non compliance is usually when a business gets into financial difficulties and they stop reporting altogether rather than stop reporting data.

In terms of the data points that are referenced in SB 755, the hours worked and the occupational code and the other elements, location, do you anticipate any significant issues with implementing that piece and obtaining the data from the employment community?

We do not. It's going to take significant marketing and outreach. It's going to take outreach to the employer community. It's going to take outreach to the employer payroll services community. It's also going to take outreach to the employer payroll program community to get them on board with being able to collect the new data elements

And the use of that standard occupational classification. That's what it's called, right. SOC coding Is that going to be difficult?

I think the states that we've learned from found that that was the most difficult item to collect, and they have added additional staff resources to assist the employer community, which we would plan to do in.

But that's sort of a startup, because if you're a business, once you figure out what your occupational classifications are, then you have it. Then you have it. So it's part of the ramp up to get the employer community on board, and chambers of commerce can help disseminate information about that I assume.

The only ongoing thing that we have is we have employers that go out of business all the time, and then we have new employers every year. So in theory, we would be educating the new employers as they come into new businesses, as they hire employees.

And I assume that while it's not true uniformly within industries, there are batches of occupational classifications that I would assume would be the same.

That's possible, but I'm not sure.

And I haven't looked at the SOC list to see what that looks like by type of industry.

Yeah, I have not referred to the SoC list either.

So in the states that are, first of all, when I was looking at the various reports that exist on this effort nationally and by state, it looks like this whole process is sort of in its infancy across the nation. Some states are in planning, pre planning, whatever that means. Were you able, when you did your survey, to determine how other states are funding this? Number one.

So all of the states that we talked to indicated it came from state funding sources.

Which, given your estimate, we should be able to do? Perhaps not this year, but on your timeline, we ought to be able to come up with $25 million out of our budget to help you provide data and make sure our training programs are more efficient.

Let's see, if we were to do this, SB 755, and we had the money tomorrow, and we could start implementing it, and you put the plan in place, how would you use these data points to change the way we optimize, we operate. We design and operate our workforce training programs, now.

This is outcome data. What would you do? What would you do that you're not doing today with respect to the way we spend money, the way we adjust the way we spend money, and the training programs that are out there in terms of how we evaluate them? Is that a fair question?

Not an easy question. I mean, like for me, for example, on the administrative side, this would result in new data elements that we think we could absorb within the Cal job. System and so on. So that's just one, you know, the first easy step, you know, in this complex thing.

Well, let me. Do you. Do we think that, that if we have these data points and we put them in play tomorrow and six months from tomorrow, we all gather here at the well and have a conversation? Do we think that the data is going to show that uniformly our training programs are training individuals for jobs that exist today, that the individuals secure those jobs, that they're regionally living wage positions, and that the positions show a career path towards something else?

I mean, it's a fair question, and I think this is why you see big commitments for performance analysis and evaluation with even the data we have. And I think it's how do we make the best decisions we have with the information we have about the programs? And I think the data elements that are suggest in SB 755 just give more information, particularly around training related employment.

But I think also recognizing in the cycle of decision making, having more data and more evidence available is very helpful at every point in time, whether it informs that program scale up or an adjustment to it or a new program. Right. And so I think the cycle of evidence based policy making is something we're very much interested in.

And I'm not being critical by my questions. I'm very much appreciative of the time and effort that went into preparing the report. Very thoughtful report, very detailed analysis, more so than my brain can possibly process. But I do recognize excellent work when I see it.

And I think this is certainly an effort that needs to continue however long it takes to implement, because whether it's $2 billion in a couple of programs, big programs, largely from the feds, plus an unknown amount of state General Fund money, which maybe that's another $1.0 billion over a period of time. Now we're talking real money at two to $3 billion.

I think we really owe everyone, including the training participants, a thoughtful and thorough analysis to make sure that participants are told upfront what the job expectations are in these programs and what they can expect to see long term in terms of career growth and wages. So I'll depend on you all to help implement that, since I unfortunately won't be around to see, see the conclusion. Successful conclusion, I hope, of this. I think you told me and answered this question. Do you anticipate that we'll be able to find any federal funding to help with this?

I would say as far as we owe a dollars, there's a. How would you say? It's very, there's a whole lot of priorities on that funding, and that was top of mind throughout, and that's why we requested state General funds. Moving forward in this budget climate, there's going to be a lot of priorities, a lot of high priority initiatives that we're currently funding and wish to continue, like services and programs that target the disabled community, be it the farmworkers and what have you.

Those are obviously very important to my colleagues, and I. I'm not suggesting that we start chopping away at programs to do this. I was just curious as to whether you thought there were any other buckets of money coming out of the Federal Government since they can print it that we haven't tapped.

Yeah, I mean, like I said earlier, I'm hoping with this national interest and this enhanced data, that one day we will see some federal funding for this type of work, but at this juncture, not on the horizon.

Okay. Well, let me see if my colleague has any questions. I do appreciate your hard work, and this is a major project and I hope it continues and you will make a huge difference in this, I think, in the State of California in workforce training, if we are able to get a handle on where the money goes, how it's being spent, how it can be spent more effectively, and you're going to help a whole ton of workers out there who are simply trying to put bread on the table. So thank you all very much.

Thank you. Let's call up our second panel to talk about the value of these new data points and what we could possibly do with information once we're able to receive it. Good morning. Thank you for joining us. Driving out here. Let me have you introduce yourselves starting to my left. You're right, ma'am.

Oh, sorry. Good morning. I'm Sarah Bohn. I'm Vice President at PPIC, the Public Policy Institute of California, where I direct our new Economic Policy Center.

Thank you for joining us, sir.

I'm Jesse Rothstein. I'm the Carmel P. Friesen Professor of Economics and Public Policy at the University of California, Berkeley, where I'm also the Faculty Director of the California Policy.

Thank you both. Feel free to start. Who wants to be first? Flip a coin, I guess.

I was first on the program, so I assumed me, but okay. Good morning. Thank you for having. For giving me the opportunity to talk with you today. I've. In addition to my current role, I've served in a variety of staff, advisory, and consulting roles at the state and federal level, advising public agencies about the collection, analysis, and interpretation of labor market data. And I'll be drawing on that experience in talking today. Let me start with the bottom line.

Enhancing the Unemployment Insurance base wage file by collecting additional information on hours worked, occupation, and the location of work as planned under SB 755 has clear benefits for California workers, employers, and the state as a whole. It would greatly expand our understanding of the California labor market and help policymakers navigate the state through better workforce, labor market, social safety net, education, and other policies.

For decades, the base wage file has played an essential role in helping policymakers assess how well our taxpayer funded workforce development programs are serving working Californians by providing timely information about quarterly employment and earnings for all workers in the state. The base wage file has served as the backbone of workforce training program evaluation in California and nationally. Data this comprehensive and this foundational to employment policy would be nearly impossible to obtain from other sources.

However, our ability to use the base wage file to support workforce training programs is limited by the coarseness of the information it contains. We can measure whether individuals are employed and how much they earn in a quarter, but there are other important metrics, such as whether a worker is working full time, whether she's found a job in the occupation for which she is trained, and whether she's earning a living wage, that we cannot measure in current data.

The California Policy Lab, which I direct, worked with the California Workforce Development Board in 2022 to study whether participation in the state's workforce training programs led to better employment and earnings outcomes. This was the Calskills evaluation that we've talked about today.

This was made possible by the integration of training program data with the base wage file under Cal skills developed under Assembly Bills 2148 and 1336, we found that many of the training programs indeed raised participants employment and earnings by a substantial amount, although the impact varied importantly across programs. Unfortunately, with the limited outcomes we had available, we couldn't do much to understand why they varied so much.

We would have liked to measure whether workers were full time or part time, whether they were working in the occupations for which they had trained, as you discussed earlier, Senator Roth, and whether they were earning living wages. With the proposed expansion of the base wage file to include occupation hours and location information, we would be able to do all of these. This would generate the kind of evidence that we need to better align California's workforce training programs with the needs of our economy.

The discussion today thus far has focused mostly on the value of an enhanced base wage file for workforce training programs, but the potential benefits that go well beyond workforce programs. So let me talk about a few of those. First, the state is now making a large investment in the cradle to career data system, which will connect information on K-12 and post secondary education with the base wage file. This will have many benefits for evidence based planning and decision making.

As one example, it will enable colleges to measure the post graduation employment and earnings of their students and thus to assess which programs are delivering the instruction students need to achieve economics success. But this work is badly hampered by the inability to tell whether Low earnings reflect part time work, partial quarter work, or Low hourly wages. An enhanced base wage file would permit such a distinction. It would also help us understand whether students are working in their field of study, another key metric for student success.

Second, the new data would permit the provision of much better labor market information for employers, workers, and planners. As a result of the COVID-19 pandemic and ongoing technological change, we're seeing structural shifts in the state's economy, with sometimes severe worker shortages in critical fields. Imagine being able to identify hotspots for high wage job opportunities in a particular occupation. We could help direct job seekers with the appropriate skills and help businesses fill their critical jobs.

Third, the new data would be useful for identifying violations of minimum wage, overtime and worker climate classification rules, and thus for targeting enforcement and compliance efforts. When some employers ignore the rules, they can obtain an undeserved advantage over other employers who are trying to compete fairly. Improving compliance in these areas will mean more workers earning living wages and a more level playing field. Fourth, data on the location of work will be enormously valuable for local economic development planning, including adjustment assistance.

Where is employment growing or shrinking? Where are we going to need to relieve transit bottlenecks or build more housing to enable continued economic growth? Where might we need to provide adjustment assistance for workers in facing declining demand for their occupations? Where might there be an opportunity to attract a new employer to take advantage of a skilled labor pool that has been freed up by changes in employment in a previous local tent pole employer?

I've spent over 25 years doing research on public policy and labor economics and over a decade working with policymakers to make better use of administrative data in service of workers, businesses, and the overall economy. My perspective is supported by decades of research that unambiguously speaks to the value of enhancing base wage information with the three data variables I described today.

My most recent study uses Census Bureau data with job locations to study mismatches between worker residential locations and locations of jobs, focusing on rust belt cities where we can get data from the census Bureau. This is exactly the kind of research on job deserts that we would like to be able to do for California, and we would be able to do with an enhanced, basically trial.

By improving our research and analytic capabilities, the state can spend taxpayer dollars for workforce training and education more efficiently, reduce costly frictions and extended unemployment driven by changes in local labor markets. Better plan for changing labor market needs, and help keep the state's economy running smoothly. Let me summarize and wrap up.

Information on workers hours and occupations will allow us to better understand the effects of changes in the labor market, on economic opportunities, and to further improve the effectiveness of our education and training system to prepare California's workforce for the 21st century. Information on location of work will allow us to better better understand who is being most affected by changes in the labor market and how we might plan for shifting economic conditions.

We know that collecting this information is achievable and cost effective because other states have done it. Without this information, our education, workforce training, and economic development planners will be half blind, unable to efficiently address new found challenges in the changing nature of work. Given the staggering impact of the pandemic on the California labor market, in particular on jobs for lower and middle income workers, and concerns about the future of work, I think these enhancements to California's UI base wage file are important and urgent.

SB 755 started the ball whirling, and I applaud Edd and the labor agency for their work in figuring out how it could be implemented. I hope that I've given you an idea of the benefits that this would bring to the state and its labor market. Thank you.

Thank you. You know, I'm sort of I had an opportunity to chair a budget Subcommitee dealing with education finance, which is a pretty large portion of our budget, about 40% for a couple years. And I was troubled by, in the higher education field in particular, we're talking about telling participants in our training programs what the opportunities are and try to link up job training with actual jobs that exist. We don't do that very well. At least we haven't, in my view, in higher education.

What's the value of a sociology degree and what can you do with that, an undergraduate degree? So I'm fascinated at the possibility that on the cradle to career dashboard, the governor's dashboard, that we could perhaps link up and be able to tell people what's out there for them when they enter University of California facility, if they choose a particular subject matter, what is out there or is not out there for them at the end of that process. And I think that's what you were talking about, sir, is it not?

Yes, that is. I think that's exactly right. Now, there's been work in the last few years on trying to document average earnings of people in various majors, but that's about all that we know thus far.

And we will be able to tell because it's a training program and there will be reports, reports on occupation once the person exits the training program, and wages earned and earned not only immediately, but over time.

Exactly. It would be possible to measure what occupations, people with sociology majors, for example, are going into, which occupations are leading to high earnings or to high earnings growth.

That could have an interesting result or impact, but I guess we'll see hotspots for job opportunities. How does that work in the context of what we're talking about with Senate Bill 755? How would we identify hotspots given the data that we're asking? Data points that were asked can be.

Reported just as an example. We would be able to see, for example, that there had been growth in the number of welders employed in a particular county over a period and that perhaps that growth was outstripping the existing training capacity for that field. That might help us to identify places where there's a need for more training capacity .

or vice versa.

Or vice versa. Exactly.

Those are the stories that tend to hit the newspapers.

Yes.

And I'm curious, with respect to workplace or labor law enforcement in the workplace, how does this, I'm trying to understand how these data points would help with that.

Well, you wouldn't want to take the, one of the things you'd be able to do with the base wage file is divide somebody's quarterly earnings by their number of hours and that gives you their hourly wage. I think there are lots of reasons why you wouldn't want to use that as gospel, but it can be a good indicator of situations where people may be being paid less than the law would require, and maybe is an area where you might want to look further.

I mean, if the data point is hours worked by days and weeks, you would be able to take a look at overtime compliance.

Overtime would be a little bit tricky, unless. Because what I believe you would get under SB 755 is the number of hours in the quarter.

So knowing how much broken down by.

Days and weeks, that's my understanding, yeah. But there would be patterns that might be suggestive of workers who are working a lot of overtime or who are working for less than the minimum wage and those. That can be the beginning of a further investigation. It wouldn't be the end of it.

Right. We didn't get more. We were not more specific. In 755, there are apparently some states that are taking a look at days and hours by days and weeks, which could provide even more information if one wanted to spend the time and go to the effort to collect it. How would this data have helped, if at all, during the pandemic? Is that a fair question?

It is. So I was one of several hundred research teams who were all racing to try to understand what was going on with the labor market in the pandemic, the work that I did, we were really trying to understand which jobs were going away, and we relied on what we called a convenience sample.

We found a payroll processor that happened to have data that we could get in a fairly short, fairly quick turnaround, that we could identify which workers jobs went away, and we were able to identify the nature of the employer, how much previous attachment the worker had, had, their hours, but we couldn't see their occupation even in those data.

I think if we'd understood which occupations the jobs had stopped by the third quarter of 2020, that would have done a lot of that, helped us a lot in planning for. How do we bring people back to work?

Because what we contemplate with SB 755 is a periodic picture of career progression. Occupations, wage increases or not, different occupation, no wage increase. So events like a pandemic or something else like a great recession would give us an opportunity to see where people are at a particular point in time.

Right. Right. As we began to come out of the pandemic, we saw that there were large sectors where employers were having trouble finding workers. We knew there were a lot of workers who had lost their jobs during the pandemic it would have impossible to use the base wage file to try to see how many workers were transitioning from the old jobs into these new high demand jobs. And that's without occupation. It's much, much harder to be able to understand that. All you can see is industry.

And you don't really know whether the janitor, you know, for example, the janitors may have lost jobs in one industry, but be perfectly suited to work as janitors in another industry. But you can't tell that if all you can see is the industry of employment and not the occupation.

And I think, I believe earlier you mentioned something about traffic congestion, mobility issues and the like when it comes to the. To employment and the workplace. Of course, I'm from Southern California, inland Southern California, where we have nothing but congestion and impact. How would these data points work to inform us about those issues and possible solutions?

I would say the most valuable data point for that set of issues is about the location of work. Right now, the base wage file reports the employer's main location. And all the workers, the only information you have about their location is their employer's headquarters. But that doesn't allow you to see if, for example, an employer is greatly expanding employment at one of its warehouses. The change in employment to that warehouse is what drives local traffic congestion.

And so being able to generate localized measures of employment growth in real time would be very valuable for understanding where we're likely to face future traffic congestion, where we need to plan for more transit or other ways of helping workers get to work.

And when I talk about our traffic situation down there in vehicle miles traveled, and the various restrictions that we have on expanding the modes of transportation, we know we're building houses out in the inland part. I just assume people are traveling to jobs in the coastal areas. But it may or may not be true. But these data points would actually pinpoint where people are working who live in particular areas of the state.

Exactly. So you could use these to generate point to point commutes.

And then finally, in some of the material that was in the binder, I note topics such as Unemployment Insurance and paid family leave being referenced as issues that these data points could impact. And maybe you could clarify that for me. How does that work?

Well, I'll just give one example. There are lots of ways that one could use this, but one example would be to understand whether all categories of workers, all classes of workers, are able to access the benefits in an equitable way. And we can do that right now by quarterly earnings to understand whether workers with higher quarterly earnings are more or less able to access workers with lower quarterly earnings.

But quarterly earnings is not a great proxy for how well off the worker is, how skilled the worker is. Hourly wages would be much better for that. And so understanding the hourly wage distribution of people who receive benefits from under different programs would be very useful.

I see. Very informative.

Thank you.

Yes, ma'am.

Thank you Senator Roth and Committee Members, for inviting me to speak with you today. Jobs in the economy are the top concern for Californians, according to the PPIC statewide survey as of February. I think the latest one was. And on top of that, Californians are very worried about economic mobility. 71% believe that the state's children will be worse off than their parents financially when they reach adulthood. And I would argue there are strong economic headwinds today.

But also, California has a lot of strengths to draw in our large, diverse, innovative economy. And for me, developing the skills, the education, the training to connect Californians to jobs of the future is critical in this regard to generate the kind of mobility that Californians are looking for and better near term outcomes. In addition, our survey work at PPIC finds that state investments in job training are among the most popular when thinking about where government can be effective.

81% of Californians say they support increased funding for job training programs. That's from our November statewide survey, and that view is shared really robustly across demographic groups, across regions, across those who identify with different political parties.

So to shed light on the value of new data collected through the base wage file, I wanted to make a few points based on my experience as a researcher studying the labor market in California, and I'll draw on research that I and other colleagues at PPIC have done using the base wage file as well. So I wanted to start with reflections on how workforce program evaluation could be strengthened by the data elements that are under discussion today.

In 2019, we published a rigorous evaluation of career education programs, or CTE programs, at California's community colleges. This was made possible under an MOU with the chancellor's office that included access to individual level, De-identified student records that were linked to elements of the base wage file.

We use this to evaluate the impact of earning a career education credential on an individual's earnings, and we found a 20% increase for graduates with a CTE associate's degree or long or short term certificate, and that's relative to what they would have earned in absence of earning that credential. This is possible to do because we have the long earnings history of CTE students, which are, on average a bit older and have long work histories compared to the kind of average college student that you might imagine.

We found, in addition, a lot of variation across programs, across types of degrees. So earnings gains were higher for the longer certificates or for an associate's degree, and across programs. So California community colleges, I know it will not surprise you, health programs, by far and away generate the largest earnings returns for graduates. That's mostly driven by registered nursing associates degrees. We also looked at, and I think this might be relevant to your questions around career progression. We evaluated career education programs that stack credentials.

So, for example, in the renewable energy space, you might start with a certificate that you can get under a year in kind of energy fundamentals, and then you might advance to the next credential, which might be a long term certificate or an associate's degree in solar installation technology. And these are programs that are really built to build in that career progression, the kind of mobility that we're hoping these programs can generate literally into the program. And there's a number of them across the state.

So we wanted to see to what extent achieving credentials along these pathways actually was reflected in earnings. And we found, unfortunately, not a lot of students who kind of start with a short term credential continue on on these pathways, which is one issue. But those who do kind of stack an additional credential, a meaningful increase in their earnings. And last, in this work, we wanted to understand what the earnings gains mean relative to kind of economic mobility to living wage that you referred to.

What we did was we looked at the kind of trajectory for students who achieved these career education credentials relative to what I would call an objective threshold of economic need that we drew from the peak PPIC Stanford California poverty measure. So it's an estimate of economic need that varies by county according to variation in cost of living. And we use that to generate what a middle income level of earnings would be for an individual.

And we found that most graduates of career education programs achieved that middle income level of earnings within six years of getting their credential. Again, varies a lot depending on the type of credential, the program, the length of the credential. So these insights, I believe, are really powerful and helpful in understanding how graduates of career education programs fare overall economically.

But they don't tell us how well the programs are preparing students for the careers in which they've trained and whether the investments that are embedded in the community colleges robust CTE offering are actually meeting labor market needs. So I want to highlight three reasons why the enhanced data that's on the table today could address those gaps.

So if we knew the occupation that these graduates of career education programs were working in, we could much better understand if the earnings trajectories that we see reflect real gains due to the training that they got. So, as Jesse mentioned, you know, in some cases, we can look at the industry and know.

So if a nursing student went from the retail sector to the hospital sector, we're pretty sure that they're working now in a health related field, but they may not be a registered nurse, even though that's their credential. And indeed, we do see some differences in gains even for that kind of high return program across demographic groups, and it could be related to under placement in the job that they've trained for.

In addition, that stackable credentials example that I gave you, which we find to be a powerful illustration of economic mobility. You know, the best that we can do is look at how earnings at the quarterly level jump, not whether those additional certificates, additional skills are actually helping somebody move into a higher level occupation in terms of work location.

You know, I mentioned that we looked at, you know, earnings relative to, you know, are you earning middle income level of wages that was done based on where you attended community college, not where your actual work is. So that's a limitation of the analysis.

Historically, in this case, community college students often work and live near where they attend school, although I think that's an assumption based on historical trends, it probably needs reevaluation because we have more online schooling opportunities and we have an intense need sometimes to move to job opportunity. So having the work location would really refine how the programs are related to how they align with regional labor market needs.

And lastly, having hourly information would allow us to be more nuanced in our analysis of the returns to these kind of credentials. So we see earnings increase. We don't know if that's because you're working more hours. And in some cases, we see, you know, not a lot of earnings growth for some programs, and that could be because your hours are muted or actually decreased for a variety of reasons.

So altogether in the space of career education, which I hope is illustrating for you how this could work in other kinds of training programs, you know, these additional data fields could add a lot of value to understanding the labor market and the value of the programs for the students and the potential employers that they may go to. I want to kind of finish on two points that a little bit more broad relative to the potential uses for, for this data in other ways.

So, first, there are a couple of other programs that the PPIC is in the midst of or has evaluated using the base wage data. And I just wanted to mention these to you in case it's part of your considerations in advancing SB 755. So my colleagues at PPIC recently analyzed the employment dynamics for calfresh participants. So this was research conducted under an MOU with the Department of Social Services, where again, De-identified individual records were linked with the base wage file.

And understanding the kind of employment realities of calfresh participants matters for understanding the needs of that program, what kind of churn those who run the program might expect. In a labor market where there's a high share of low wage jobs, there's often a lot of volatility in work for low income populations. The additional another team at PPIC is evaluating the effectiveness of Department of Corrections and Rehabilitation programs, again using De identified individual level data linked with the base wage file.

So in both of these cases, you know, enhanced data on the occupation hours and location of work could really provide for a more detailed labor market analysis to inform the efficacy of those programs that relate to kind of labor market outcomes for Californians who touch those state programs. And finally, just generally, you know, I believe these data enhancements can help California meet the labor market challenges that lie ahead.

Given the transitions that are on the horizon from climate change, then transition to clean energy, from AI and automation, to major demographic changes, including the retirement of a very large baby Boomer generation, our workforce systems are going to need to be adaptable and accessible. And while we don't want to burden employers with additional data reporting requirements, we do want to ensure that employers can rely on a trained, available workforce where they can confidently locate in California and grow.

Other states and national organizations are looking to enhance this data. As we've talked a little bit about today, you know, with the aim of strengthening the link between education and opportunity, and at a time when Californians are very concerned about the economy, both now and for the future for their children, I'm really encouraged by the potential to strengthen the state's ability to shepherd successful strategies for economic growth and economic well being for our families, our businesses, and our communities. Thank you so much for the opportunity to be here today.

Thank you for those comments. With respect to the community college CTE program evaluation, I gather then as a result of the evaluation, you were not able to actually established that individuals trained were working in jobs for which they were trained.

That's exactly right.

So you have the same, you had the same limitations in your PPIC evaluation and study that all the rest of us have exactly. We don't know, for example, whether a CNA or an LVN that comes out of a training program and reports a wage increase. Whether the wage increase is because the LVN or the CNA took a job at Nordstrom's or Mcdonald's or something else.

For example, we're pretty sure that there is a bump from getting the credential because what we see is a trajectory of earnings that is pretty consistent at a, you know, a rate that we can identify statistically. And then after a credential, you see just kind of the trajectory changes. It kind of bumps you to a higher level that's in an effective credential that is earned.

But we can't say for sure if it was exactly that credential or something else about what changed in that student's trajectory at that time, that they got their degree.

Or the General labor market or the economy or something we did up here with respect to required wages.

Wages, exactly.

Are there, has anyone been able to do a study or evaluation of training programs, for example, either narrow or broad, to actually connect the dots as to whether or not individuals trained received jobs in the areas for which they were trained?

Well, I'll let Jesse speak to this, too, but there is a robust academic literature on job training programs that, for the most part, I think it's fair to say, uses different kinds of data that's more detailed that actually either it's a study that is tracking, sometimes at high cost participants before, during and after. So it can gather a wide range of information about them with which to kind of then back out the impact of a program.

A lot of these, these evaluations are done when it's feasible, given data or given the opportunity to do a controlled kind of trial of a program. So it may be hard to generalize from those individual studies on kind of the impact of training programs. For instance, at work in California today.

It appears to me that it actually would depend on someone reaching out and touching a participant in a program, having the participant being willing to provide information and then collecting this voluntarily supplied information and trying to connect dots, which I guess would call into question the validity of the study in terms of who participates and who doesn't. Is that about right?

I think that's right. When this is done, it's typically done as part of large multimillion dollar evaluations funded by the Department of Labor, the US Department of Labor that rely on surveys. They survey a sample of people, they try to get people to participate and they follow up with them after the training. That makes it very expensive. It means you only do it every 10 or 15 years.

It also means that the samples tend to be fairly small, which one of the consequences of that is that they can often evaluate a General class of training programs, but they can't get much texture into particular programs or particular credentials that people might be studying for. That's just infeasible to do that with surveys.

I remember when I was dealing with education finance, I'd asked, particularly in the higher education area, can't you tell me what students, what your graduates are doing at the two year point and at the five year point? And the answer was no. We have to, we'll send a survey out through our alumni Association. So to those students who've thought enough about their school to stay in contact with the alumni Association, they may or may not respond.

But we can't collect data with any degree of accuracy and in detail to draw conclusions as to what students are doing, doing how they're doing, and whether the job that they have, assuming they have one, matches up with the degree and the training program that we provided in the college or University in question. And, you know, given the amount of money that we pass through the pipeline here in the State of California, pick a figure, 2 billion, 3 billion, 4 billion.

Over a period of time, we ought to have a better data collection effort and a better analysis system going, going on so we can validate the training programs or not based on how effective they are in meeting the needs of the participants. Well, you all have been very, very informative. Senator Ochobog, any questions or comments? Thank you all for coming and for participating and presenting excellent testimony.

Let's see if anybody in the room wants to provide public comment. I don't see anybody jumping up. I see heads nodding, which is no. So I think we're getting close to closing the hearing out. I want to thank the Administration, Governor Newsom, for making Administration staff available to participate today in the oversight hearing and, frankly, for authorizing their participation in the preparation of an excellent report on Senate Bill 755.

I want to thank you two for traveling from wherever you came from to be with us today with some very informative testimony. And again, I want to thank the gentleman to my left, Dan Rounds, and my staff, Carney King. And I see my Chief of Staff in the hearing room, Gilbert Martinez, for their help in pushing through Senate Bill 755 and for putting together the information necessary for this oversight hearing. More to come, I hope.

I see a Bill next year from my vantage point, wherever that is directing in some way on some basis, the employer community to start over time submitting data and authorizing the appropriate appropriation to the various departments and agencies to put this 755 in play. Stay tuned. Thank you all very much for attending. Have a great day. The hearing of this Select Committee is adjourned.