Using utilization-focused evaluation in policy decision-making: Community-engaged design, process, and implementation

Abstract Information: Evaluations can be useful only when they are accessible to the intended users. Most evaluations of publicly funded education programs are conducted as summative studies with the goal of assessing a funded program’s effectiveness. Traditional methods encourage the use of experimental and quasi-experimental designs to compare outcomes of participants and a comparison group, often producing written reports and articles as guidelines for recommendations. While this approach is widely accepted, the application of data to meaningful decision-making about program implementation has been largely limited to those with greater academic literacy. Those outside of evaluative research or the academic community often face barriers to usable information such as outdated static analysis, difficulty finding free and reputable sources, and complicated jargon. Our evaluation for Michigan’s early childhood and out-of-school time programs added a different approach. Following principles of community-engaged scholarship, our evaluation has focused on the use of data for continuous improvement and decision-making, in an approach aligned with Patton’s (2011) utilization-focused evaluation (UFE) framework. We have listened to and collaborated with the intended users of our evaluation to produce data they can use immediately to improve program quality, better facilitate equitable access, and guide policy decisions. Using big data—not only program and school student data but also community data such as census information and Child Opportunity Index scores—we produced a series of online maps and data dashboards that display information in ways program leaders and policymakers can easily understand. Policy-specific visual tools can provide easy and interactive summation of information, encouraging wider use of data outside academic spaces. Visualizing data at state, regional, and local levels offers a new perspective on available data, as unforeseen regional differences and similarities emerge. This presentation explains how our work with Michigan’s early childhood and out-of-school time key stakeholders follows the steps of UFE. We will use brief case studies to demonstrate the responsive and iterative nature of the work. We conclude with implications, emphasizing the ways in which evaluators can capitalize on community expertise, and data beyond child outcomes, to facilitate vital policy and program decisions.