Evaluating a Multi-Institutional Alliance Clinical and Translational Science Award: The Integrated Translational Evaluation Model (ITEM) for outcome, benefit, and ecosystem impact evaluation

Abstract Information: The complexity of evaluating the activities and achievements of a Clinical and Translational Science Award (CTSA) becomes more challenging as the CTSA moves into an equitable multi-institutional alliance structure. The University of Texas Medical Branch at Galveston has created such an alliance with three other local institutions (Texas Southern University, Houston Methodist Hospital System, University of Houston Clear Lake). This alliance brings diverse perspectives to the CTSA team and diversifies student and community populations impacted by CTSA activities. The CTSA alliance model requires an evaluation model that can manage these complexities of an alliance structure and provide rigorous evaluation with the flexibility of relevant approaches and tools that can create a contextually appropriate framework for process and outcome evaluation. The integrated translational evaluation Model (ITEM) offers a framework for evaluating diverse areas of translational science, from workforce development to informatics, and key facilitators such as DEIA and Team Science. This presentation will examine ITEM, discussing how it builds upon our previous evaluation successes with logic models (1) , the cause-and-effect metrics of the Balanced Scorecard (2) , and CQI (3) , while adopting recent evaluation advances for clinical and community health impacts derived from the Translational Science Benefits Model (4), equity-based feedback evaluation (5), and measurement of longer-term collective impacts of the ecosystem via Social Benefits Accounting (6) . A unique advantage of ITEM is that it connects evaluation components (e.g., metrics) to higher-order components (e.g., vision, mission, and strategic goals), a common challenge in evaluating translational science programs. This model gives exposure to multiple methods in an integrated approach that can guide young evaluators in acquiring complex evaluation expertise. It also ensures the centering of all voices and creates an evaluation model that can ensure that the diversity of voices related to alliance based CTSA work are amplified. References 1) Knowlton LW, Phillips CC. The Logic Model Guidebook: Better Strategies for Great Results. Sage; 2012. 2) Kotarba JA, Wooten KC. The innovation scorecard for continuous improvement applied to translational science. Journal of clinical and translational science. 2017 Oct 2017;1(5)doi:10.1017/cts.2017.297 3) Loper AC, Jensen TM, Farley AB, Morgan JD, Metz AJ. A Systematic Review of Approaches for Continuous Quality Improvement Capacity-Building. Journal of public health management and practice: JPHMP. 2022 Mar-Apr 2022;28(2)doi:10.1097/PHH.0000000000001412 4) Searles A, Doran C, Attia J, et al. An approach to measuring and encouraging research translation and research impact. Health research policy and systems. 08/09/2016 2016;14(1)doi:10.1186/s12961-016-0131-2 5) Ramos G, Ponting C, Labao JP, Sobowale K. Considerations of diversity, equity, and inclusion in mental health apps: A scoping review of evaluation frameworks. Behaviour research and therapy. 2021 Dec 2021;147doi:10.1016/j.brat.2021.103990 6) Bornmann L. Measuring the societal impact of research: research is less and less assessed on scientific impact alone--we should aim to quantify the increasingly important contributions of science to society. EMBO reports. 2012 Aug 2012;13(8)doi:10.1038/embor.2012.99