Measuring Infrastructure Changes and Institutional Impact

Abstract Information: Systems change initiatives present challenges in attributing outcomes to program strategies and activities. Yet, funders and program leaders alike depend on evaluation to understand the extent to which investments have led to sustainable infrastructure changes. This session presents a model for evaluating infrastructure changes and the institutional impact of system change initiatives. The model was initially developed to support the evaluation of a multi-site, statewide NIH-funded Clinical and Translational Research (CTR) program. Effective evaluation of infrastructure changes, institutionalization, and likelihood of sustainability assumes that we, as evaluators, know what the institutionalization of a program looks like. In essence, we are predicting what program processes and components will still be operational, tangible, and visible when program funding expires. This session explains the development of a model to measure infrastructure change and institutional impact, and shares the initial model that was developed. Model development was based on a review of literature, documentation from organizations regarding how they define the institutional impact of a program, and investigator case studies examining individual career impacts and perceived institutional changes of a program. The resulting model includes four dimensions: infrastructure, recognition, partnership, and ecosystem. Evaluation of the infrastructure dimension focuses on institutional enhancements, operating or existing without program funding, stemming from program investments. For the CTR program, this includes new centers or institutes, equipment, collaborations and partnerships, training programs, procedures and protocols, and policies. Recognition is assessed based on changes in institutional rankings, publications and grants stemming from program investments, return on initial program investment, and individual investigator recognition and awards. Partnerships refers to inter-institutional agreements and collaborations created as a result of the program, yet extending beyond the program, including research collaborations, as well as community partnerships. Finally, assessment of the ecosystem dimension involves the examination of changes beyond infrastructure that have created a culture of continual growth and creativity. For the CTR program, an ecosystem would be one in which investigators work to develop other investigators who work to develop new investigators – creating a cycle, or an ecosystem, that is self-perpetuating. In addition, in an ecosystem, investigators at the organization or within the system are routinely invested in to become researchers, mentors, and leaders; the sharing of expertise and mentoring of junior staff are rewarded; and policy and culture changes, learning systems, mentoring cycles, and leadership pipelines created through the program lead to continual growth and productivity. Measurement of these four dimensions is based on multiple data sources, including institutional and program data and interviews with key informants. This session will be interactive, with feedback sought from attendees to inform continual model refinement and improvement. It is hoped that with continuous improvement, this model will be useful to evaluators in understanding the important dimensions reflecting institutionalization and consequently in assessing the degree to which investments in systems change initiatives are likely to be sustained.