Unraveling Complexity: A Mixed Methods Approach to Evaluating Policy Change in Multi-level Systems

Abstract Information: Acknowledging that health disparities stem from deep rooted structural and systemic racism, this demonstration aims to showcase how evaluators can adopt a Systems Change Evaluation Framework to assess the effectiveness of interventions that impact intricate and dynamic systems. There are many ways to define “systems” and for the purpose of this demonstration, we use this definition by Coffman (2007) “A system is a group of interacting, interrelated, and interdependent components that form a complex and unified whole. ” Specifically, this demonstration focuses on a hospital system and provides a thorough examination of the methodological approach required to evaluate a recent initiative conducted by the New York Academy of Medicine (NYAM) on behalf of the New York City Department of Mental Health and Hygiene (NYC DOHMH). This project involved devising a systems change evaluation strategy to investigate the impact of removing race adjustment from clinical equations in under resourced hospital settings and how this impacts access to and use of hospital services by African American/Black individuals. Clinical equations using race adjustment often negatively impact the diagnosis, treatment, and care for people of color, perpetuating racial biases and disparities within medicine. Systems change is defined as “any change to a system which improves outcomes for the intended beneficiaries of a system, is sustainable in the long-term, and is transformational” (Mayne, 2015). Additionally, this demonstration will also highlight how systems evaluation differs from traditional program evaluation. Given that systems change evaluations are complex and require multi-pronged approaches, and there is no “one size fits all” approach, we demonstrate how we employed mixed methods including process, outcome and economic evaluations using the Healthy Equity Frameworks. Additionally, the evaluation is based on the complexity theory which recognizes that initiatives such as ours are affected by many factors both internal and external to the intervention. We also demonstrate our logic model which links the initiative inputs including resources to activities and to expected multi-level individual, community, organizational and policy outcomes in the short, inter-mediate, and long term. Lastly, we will demonstrate how to utilize multi-pronged analytical approaches, including complementary quantitative and qualitative research methods, to evaluate system and policy initiatives including translating findings into actionable policy.