71 - The challenge of using standard evaluation approaches that are not fit for purpose: A case example from the NIH-funded Clinical and Translation Science Award program.

Abstract Information: In this roundtable, we reflect on our journey in evaluating a Clinical and Translational Science Award (CTSA) program and describe potential best practices of CTSA evaluation teams doing similar work. We begin by discussing the NIH’s CTSA structure, noting that it meets the criteria for a complex intervention. We then share our experience applying one standard evaluation approach, which is to use logic models. We will detail our experience of making sense of program-level logic models with the CTSA program and executive leadership to identify operational gaps and opportunities for improvements to the efficiency and effectiveness of the CTSA considered as a whole. Panel Discussion Question #1: Have any other evaluators encountered a similar situation where they were asked to apply logic models for evaluating a complex intervention? If so, how did you develop your logic models? What concerns did you have about using logic models for this purpose? Following this discussion, we will share how the paths of the two presenters crossed and how this cross-fertilization led to a realization that the standard use of logic models was not fit for purpose; that is, the evaluation approach did not match the CTSA design characteristics. We share the limitations of the standard approach and why an evaluation approach rooted in system thinking is a better fit for evaluating a CTSA. We conclude by sharing barriers and facilitators to getting the CTSA program and executive leadership to adopt an evaluation approach not prescribed by the funder, and provide practical recommendations for evaluations of CTSAs. Panel Discussion Question #2: What challenges have you experienced, or do you foresee, in advocating for an evaluation approach grounded in systems thinking? How do you work with leadership to engage different evaluation approaches?