Chairperson Bonds and members of the committee, thank you for the opportunity to testify today. My name is Caitlin Schnur, and I am the Deputy Policy Director at the DC Fiscal Policy Institute (DCFPI). DCFPI is a nonprofit organization that shapes racially-just tax, budget, and policy decisions by centering Black and brown communities in our research and analysis, community partnerships, and advocacy efforts to advance an antiracist, equitable future.
My testimony focuses on the Marion Barry Summer Youth Employment Program (MBSYEP) and the of funding and implementing a rigorous evaluation of the program. Every year, MBSYEP connects about 10,000 young residents to entry-level subsidized summer jobs, helping them to build work experience and putting money in their pockets. Nearly nine in ten of those benefitting from the program are Black, illustrating how the program plays an important role in countering the harm of systemic racism and advancing racial equity in employment and economic opportunity in the District.[1]
While DC has improved its evaluation of this program in recent years, lawmakers should require the Department of Employment Services (DOES) to contract with a third party vendor to implement a rigorous evaluation of MBSYEP. A rigorous evaluation can show if there is causal evidence that the program is effective at advancing employment, education, training, and other related outcomes for the District’s young workers.[2] Understanding MBYSEP’s efficacy is important given DC’s glaring racial disparities in employment: from 2021 to 2023, the average unemployment rate among young Black workers, ages 20- to 24-years-old, was 16.1 percent, compared to just 3.1 percent for their white peers—a ratio of about five to one.[3]
The committee, DOES, and other workforce development stakeholders can use the results of the rigorous evaluation to take steps to course correct program approaches if necessary and invest more in the most effective aspects of the program. Ensuring that MBYSEP is moving the needle on employment-related outcomes for young Black workers can also help advance racial and economic justice in the District.
Current MBSYEP Evaluations Do Not Generate Robust Evidence for the Program’s Efficacy Relative to Employment, Earnings, and Other Related Outcomes
MBSYEP is the largest summer youth employment program per capita in the country, serving more than 14,000 young people in 2023.[4] In 2024, the program’s 45th year, MBSYEP anticipates serving 10,000 young workers.[5] MBSYEP also represents a significant investment of public dollars: the approved fiscal year (FY) 2024 budget included $27.3 million in spending for the program.[6] Additionally, the lion’s share of MBYSEP participants are young Black workers, who experience outsized rates of unemployment. Unemployment rates by race show that, overall, Black workers in DC experience chronically higher levels of unemployment and are much more likely to be underemployed than white workers.[7] These data point to the harm of centuries of anti-Black public policy and ongoing structural barriers to work for Black residents, including young people, and the critical need for effective workforce development.[8]
Despite MBYSEP’s size and cost, and the magnitude of racial inequity in the District, existing law does not require DOES to rigorously evaluate MBSYEP. Instead, the law requires an annual independent evaluation of the program using methods that include 1) a pre- and post-program survey of participating young workers and employers and 2) interviews with youth workforce development stakeholders, experts, and providers.[9]
The existing evaluation methods and criteria yield some useful descriptive information about MBSYEP. The independent evaluations contain participant demographic data and offer insights into young people’s and employers’ experiences with different aspects of program implementation.[10] For example, among other program components, the evaluation must assess youth and employer satisfaction, the percentage of young people who participated in various program activities, and young workers’ “sense of progress” in their job readiness and work skills.[11]
However, existing evaluations fail to generate strong evidence for the program’s efficacy relative to standard workforce development outcomes such as employment rates, median earnings, credential attainment, or skill gains that support a young worker’s career advancement.[12] The US Department of Labor, for example, requires that youth employment programs receiving federal workforce dollars assess their program performance based on these and other outcomes.[13]
While MBYSEP is not accountable to federal workforce metrics, rigorously evaluating the program relative to these outcomes would help DOES determine whether MBYSEP participants are better prepared for work, and , than their non-participant peers. Additionally, evaluating MBYSEP relative to US Department of Labor workforce outcomes would bolster alignment across the District’s youth public workforce system, allowing the committee, DOES, and other stakeholders to better understand how the youth employment ecosystem is performing overall.
Rigorous Evaluation Can Establish Causal Evidence for Program Outcomes and Help Stakeholders Scale What Works
Without a rigorous evaluation of MBSYEP’s employment-related outcomes, the committee, DOES, and workforce development stakeholders cannot accurately gauge the program’s efficacy in helping young workers achieve career success. Investing in a rigorous evaluation of MBSYEP can build a strong evidence base for the program’s effectiveness relative to employment, education, training, and other related outcomes for the District’s young workers. It would measure participant outcomes and establish whether those outcomes are the result of the program rather than random chance. Depending on its design, a rigorous evaluation may be able to identify which program components, or combination of components, are most beneficial to participants.[14]
Evidence-based evaluation data are critical for the committee, DOES, and other workforce development stakeholders to understand how well the program is working, make necessary course corrections to improve outcomes, target services and supports to young workers, and scale what works. Within the context of MBYSEP, which overwhelmingly serves young Black workers, having the data to drive these types of programming decisions is a matter of racial equity. Finally, the committee and DOES can also leverage evidence-based program outcomes to increase resources for the MBYSEP program or to support program replication in other localities.
Program Evaluation Experts Can Support the Design and Implementation of a Rigorous Evaluation
Because rigorously evaluating MBYSEP would be a new and complex undertaking, the committee and DOES should partner with program evaluation experts to identify appropriate and ethical evaluation methods. The gold standard for a rigorous program evaluation is a randomized control trial (RCT), or an experimental study, in which eligible young workers would be assigned, at random, to either a treatment group or a control group. The treatment group would participate in MBSYEP, and the control group would not. By following both groups of young people over time, this type of evaluation would be able to determine if program participants’ outcomes can be attributed to MBSYEP.
In the case of evaluating MBSYEP, an RCT may not be the appropriate methodological approach, because the young people randomly assigned to the control group would be denied program access over the course of the study. Other quasi-experimental methods, which do not rely on random assignment but can still yield strong evidence about program efficacy, may be a more appropriate rigorous evaluation method for MBSYEP.[15]
It is beyond the scope of this testimony to recommend which rigorous evaluation methodology will work best for assessing MBSYEP’s outcomes and impact. However, by working with program evaluation experts, the committee and DOES can determine how best to design and run a rigorous evaluation. This type of technical assistance and support can help ensure the success of any evaluation.
Thank you for the opportunity to testify. I am happy to answer any questions you may have.
[1] DCFPI analysis of self-reported demographic data in the 2018 to 2022 independent evaluations of the Marion Barry Summer Youth Employment Program (MBSYEP).
[2] Urban Institute, “Quantitative Data Analysis – Experiments,” accessed February 15, 2024.
[3] Economic Policy Institute analysis of Current Population Survey microdata from the US Census Bureau.
[4] District of Columbia Executive Office of the Mayor, “Mayor Bowser Highlights Pathways to Health Careers During Mayor Marion S. Barry Summer Youth Employment Program Launch,” January 22, 2024.
[5] DC DOES, “2024 MBSYEP Program Overview,” accessed February 15, 2024.
[6] FY 2024 Approved Budget and Financial Plan: Department of Employment Services Budget Chapter, July 14, 2023.
[7] Caitlin C. Schnur and Erica Williams, “DC’s Extreme Black-White Unemployment Gap is Worst in the Nation,” DC Fiscal Policy Institute, July 26, 2023.
[8] Doni Crawford and Kamolika Das, “Black Workers Matter: How the District’s History of Exploitation & Discrimination Continues to Harm Black Workers,” DC Fiscal Policy Institute, January 28, 2020.
[9] DC Official Code, “§ 32–245. Evaluation of the summer youth employment program,” accessed February 15, 2024.
[10] See, e.g., The Coles Group, “2019 MBYSEP Independent Evaluation,” March 6, 2020; The Georgetown Firm, “MBSYEP Evaluation 2022,” September 21, 2023.
[11] Ibid, § 32–245(b)(4).
[12] Regardless of whether DOES pursues a rigorous evaluation of the MBSYEP program, tracking and reporting employment-related outcomes is especially important given that the MBSYEP program serves young workers ages 22- to 24-years-old, who are likely seeking competitive employment or further training following program completion. A 2016 audit of the MBYSEP program led by the Office of the District of Columbia Auditor (ODCA) concluded that the employment outcomes data reported DOES had large gaps and “should be viewed with caution” (pg. 35). The audit recommended that DOES strengthen its ability to track MBYSEP employment outcomes, especially for older youth. It is not clear to DCFPI if DOES has done so. None of the independent MBYSEP evaluations from 2018 through 2022 reported on post-program employment outcomes for young workers ages 22- to 24-years-old. To learn more, see: Ronald Gaskins and Jason Juffras, “Review of Marion S. Barry Summer Youth Employment Program Data and Activities,” Office of the District of Columbia Auditor, June 2, 2016.
[13] Employment and Training Administration, US Department of Labor, “WIOA Performance Indicators and Measures,” accessed February 15, 2024.
[14] Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman, “Evaluation: A Systematic Approach (Seventh Edition),” Thousand Oaks, CA: Sage Publications, Inc., 2004.
[15] Mark Courtney and Fred Wulczyn, “Quasi-Experimental Evaluation Designs,” OPRE Report 2021-114, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2021.