170 - Cross-site Variation in the Impact of Education Programs: Empirical Evidence from Multi-site and Multi-level Evaluations

Abstract Information: This study empirically estimated the cross-site variation in program impact in a set of multisite cluster randomized trials (MSCRTs) evaluating the efficacy of education programs and examined the factors that may influence the magnitude of the variation. The factors included program intensity, program type, outcome domain, and targeted grade level. The analyses were conducted with restricted-use data from completed large-scale MSCRTs funded by the Institute of Education Sciences (IES). The study used hierarchical linear models to estimate cross-site impact variation and multiple regression to explore the association between program/study characteristics and the magnitude of cross-site impact variation. The findings from this study suggest the magnitude of cross-site variation in treatment effects ranges from 0.09𝜎 to 0.37𝜎. The treatment effects of computer-based interventions and interventions targeting elementary school students are found to vary more substantially across sites than non-computer-based interventions and interventions targeting other grade levels. Implications of the results to the research, policy, and practice of future MSCRTs in education were discussed.