Approaches to Selecting Comparison Groups in Evaluations of Education Programs Implemented in Schools

Abstract Information: Evaluations of education programs funded by federal and foundation sources have increasingly required inclusion of studies capable of producing promising evidence of effectiveness of strategies being implemented. These are often referred to as impact or “evidence of promise” studies. Randomized control trials are the gold standard for such causal inference. Funded projects of education initiatives often include small numbers of schools identified based on need and desire to participate with implementation in the whole school. In these situations, random selection of schools to treatment (implementation of strategies under the funded project) and control conditions is not feasible. Thus, most evaluations of these projects rely on quasi-experimental methods. We serve as evaluators on multiple evaluation projects that require these types of studies. Our presentation will share examples from our work on how we have identified comparison groups and analyzed data for evaluations of education projects. 
Our examples are from evaluations of funded projects under US Department of Education programs that require “evidence of promise” studies. These include the Magnet Schools Assistance Program (MSAP) and the Statewide Family Engagement Centers (SFEC) program. The “evidence of promise” studies for both programs focus on whether academic achievement improves more for students in participant schools than for students in comparison schools. For the MSAP project, we will describe a two-stage matching process used by an assessment contractor for identifying a Virtual Comparison Group (VCG) of schools and students for comparison with the MSAP schools and students. For the SFEC project, we will explain a propensity score matching process used to identify a set of matched comparison schools using publicly available data from the state’s department of education. Subsequent outcome analyses for these “evidence of promise” studies will also be described. Both methods to identify matched comparison groups attempt to identify schools with comparable demographics and baseline achievement as the students participating in the funded projects. Our presentation will emphasize both the strengths and limitations we encountered when utilizing these approaches.