Developing a New Way to Measure Evaluative Thinking

Abstract Information: Evaluation scholars and practitioners have argued that having more people who can think evaluatively is “essential” for successful evaluation practice, organizational improvement, and even healthy democracies. However, empirical evidence backing those claims is scant. Making a case for investing more resources into building people’s capacity to think evaluatively requires a better understanding of its impact and added value. We need a way to measure it. While two scales to measure evaluative thinking exist, both have shortcomings and neither has been used to build a body of evidence concerning the contributions of evaluative thinking to the evaluation field and beyond. A two-stage, mixed methods study was conducted to create an improved evaluative thinking scale. Stage 1 further clarified the dimensions underpinning evaluative thinking through a literature review and interviews with leading scholars in the field of evaluation. Stage 2 generated items that best probe the dimensions identified in Stage 1 and refined those items through further communication with the original sample of scholars. Face validity was determined through focus groups with a sample of graduate students; and finally, the refined items were administered to a larger random sample of 250 random participants. A factor analysis determined this scale probes six factors with at least three items for each factor loading at 0.48 or higher and a test of Cronbach’s alpha determines it is reliable (α = 0.96). This study, conducted by a PhD student in Evaluation and Applied Research, enhances our capacity to measure evaluative thinking and lays the groundwork for future research into understanding its impact more deeply. This paper presentation will take the audience through the methodology, results and discussion of the paper, revealing the six factor structure which emerged. The presentation will also emphasize the study's strengths and limitations as well as explore exciting areas for future research.