Skip to main content
Evaluation 2024 Main banner
Icon Legend
Presentation Icons
Birds of a Feather
Birds of a Feather
Pres Strand
Pres Strand
Extra Cost
Extra Cost
Ignite
Ignite
TIG Multipaper
TIG Multipaper
General/Social
General/Social

Paper

Research on Evaluation

Session: Innovations in Evaluation Theory, Methods and Models

Findings from a systematic review of a half-century of transdisciplinary meta-evaluation practice: meta-evaluation and the general logic of evaluation

Stream: Evaluation Foundations and Methodology

Thursday, October 24, 2024
4:00 PM - 4:15 PM PST
Location: C125-126
Abstract Information: Meta-evaluation stands as a method of quality assurance for evaluations and a form of peer-review of evaluators. The explicit practice of meta-evaluation dates to Scriven’s 1969 naming of the concept. Since then, it has taken on a full life, some claiming it to have entered a period of maturity since 2010 when the Joint Committee on Standards for Educational Evaluation published a new edition of program evaluation standards that included evaluation accountability (meta-evaluation) as a standalone quality criterion. While much has been written to guide practice, no comprehensive study has looked at practice reports to assess how well practice aligns with such guidance. To address this absence, we systematically gathered a corpus of meta-evaluation practice reports (n = 174), developed an extensive data extraction and coding protocol, and analyzed this unique set of reports. As a foundational quality check, we asked if the vitally important general logic of evaluation was reflected in this corpus of meta-evaluation practice reports. We found that 82% explicitly reported criteria of merit against which the meta-evaluand was observed, 58% explicitly reported standards of achievement, and 91% provided judgments on the meta-evaluand as a direct result of the meta-evaluation. In this presentation, we unpack these numbers and discuss the value of the general logic of evaluation in understanding and describing meta-evaluation practice. Our systematic review uniquely illustrates the first half-century practice of meta-evaluation and lays out a path for its improvement in the next. New and emerging evaluators will find this presentation useful in understanding the general logic and how it “rolls up” to meta-evaluation practice and be prepared to participate in the important discussions about our discipline’s future. Furthermore, the concept for this systematic review grew out of the interests of students enrolled in a graduate meta-evaluation course and the presentation will also share lessons learned from mentoring emerging evaluation researchers through a research on evaluation project and how this collaboration enhanced the research. As evaluation quality has direct consequences for program stakeholders and society at large, understanding the mechanism of improving that quality, meta-evaluation practice, is pivotal for professionalizing the transdiscipline and sub-fields of evaluation.