24 - Why are Costs and Benefits so Rarely Adopted in Evaluation? Learning from the Diverse Voices and Experiences of Evaluators

Abstract Information: Within the evaluand’s context, its resources, capacity, time, culture, (Persaud, 2018; Persaud & Dagher, 2020; Persaud & Yates, 2023) and infrastructure (Bustamante Madsen et al., 2018) influence the conduct of cost-inclusive studies. Yates (2018) identifies the greater time and effort such evaluations require, whereas Persaud (2018, 2020) argues that many evaluators do not possess the requisite skills to conduct such evaluations, concluding that many do not understand the basics of cost-inclusive evaluations or lack the expertise to undertake such studies. It has further been argued that problems have been encountered when collecting data on outcomes, costs, and benefits, with Yates (1979, 2018) critiquing the tradition of making assumptions in cost-inclusive studies. And Persaud and her colleagues highlight challenges associated with determining costs and benefits in addition to double-counting, and incomplete or missing records that make it challenging to conduct cost-inclusive evaluations (Persaud, 2020; Persaud & Dagher, 2020; Persaud & Yates, 2023). Also problematic are the timing of such evaluations, i.e. whether a cost-inclusive study is conducted ex-ante or ex-post (Persaud, 2018). Drawing on an exploratory qualitative methodology, this study developed a deeper understanding (Creswell & Poth, 2018; Maxwell, 2005) of issues affecting evaluation capacity to conduct and use cost-inclusive evaluations (Milstein et al., 2002; Stevenson et al., 2002; Stockdill et al., 2002). Using a phenomenological approach (Creswell & Poth, 2018) with a rich, saturated criterion sample (Goertz & Mahoney, 2012) of 23 professionals who self-identified as evaluators and had in their roles as principal investigators conducted at least one evaluation study on cost analysis, cost-effectiveness analysis, or benefit-cost analysis, the study included diverse voices in various international sectors and contexts. Participants also represented a variety of evaluation roles in funders, evaluands, academia, and evaluation. Preliminary findings identify critical factors that influence the conduct of cost-inclusive studies and may affect their quality. By organizing these factors into three buckets, the study reveals issues within the evaluand’s context, the evaluator’s competencies, and the evaluator’s use of methodology that impact the quality of cost-inclusive evaluations. The study expands on the factors associated with the poor uptake of cost-inclusive methodologies and identifies strategies that may help to off-set these challenges. Additionally, findings point to both demand and supply factors that should be considered before conducting cost-inclusive evaluations (Boyle et al., 1999). Specifically, this study recommends that evaluability assessments be to undertaken to identify factors within evaluands’ contexts or evaluators’ competencies and intended use of the different cost-inclusive methodologies that may hinder the successful completion of a cost-inclusive evaluation (Wholey, 2002). Such an assessment of readiness will determine the viability, cost, and benefits associated with a cost-inclusive evaluations (Persaud & Yates, 2023), improve the conceptualization of such studies, which has implications for determining their cost (Persaud, 2018), and support the development of proactive strategies that identify strengths, address weaknesses, reduce challenges or put counter measures in place before the evaluation commences.