172 - A collaborative approach to evaluating a collective impact initiative: The Carolina Across 100 “Our State, Our Work” Program

Abstract Information: Carolina Across 100 (CX100), is a pan-university effort to partner with communities in each of North Carolina’s 100 counties to address longstanding challenges exacerbated by, and new challenges associated with, the COVID-19 pandemic. The first program in this initiative, Our State, Our Work (OSOW), aimed to connect Opportunity Youth (OY) to living wage employment and educational opportunities through the partnership of 13 community collaborative teams, representing 37 counties across the state. The structure of OSOW is informed by a modified Collective Impact Model (CIM). Through its core tenets and guiding principles, the CIM provides a framework that emphasizes the integration of diverse voices from a range of industries and positions as they collaborate on a shared goal. The purpose of our evaluation was to demonstrate how the CIM enhanced cross-sector collaboration to promote large-scale change. Evaluation Design Our mixed-method evaluation consisted of 1. a process evaluation to assess the implementation of the OSOW program, and 2. an outcomes evaluation to assess the proximal effects of OSOW on the target communities. The evaluation approach was informed by the adapted CIM and further supplemented with community engagement and social capital lenses. The adapted CIM is composed of four core components: 1.) a common agenda, or common understanding of the problem being addressed, 2.) a backbone infrastructure composed of dedicated staff to coordinate across participating organizations, 3.) metrics in form of a Shared Measurement System (SMS) to collect data consistently across participants, and 4.) mutual reinforcing activities coordinated across a common plan of action. Evaluation Activities Evaluation activities included monitoring content and structure of project-wide meetings; developing and assessing shared understanding and programmatic impact across collaborative teams; measuring implementation and reach of aligned efforts across teams; and satisfaction of support provided from the backbone organization. Given the focus on collaboration, evaluation activities engaged the collaborative teams as meaningful participants. Collaborative teams co-developed the SMS to reflect the shared priorities of community members and organizations within the program. The evaluation team utilized ongoing feedback loops with collaborative teams and the OSOW program team to inform program improvement by regularly disseminating evaluation findings. Results Our evaluation findings underscore collaboratives’ satisfaction with the OSOW program, particularly related to resources shared and support provided by the CX100 program team. Findings emphasize the value of activities aimed at fostering connection and collaboration among collaboratives. Opportunities to network, share information, and dedicated time for team planning were key program strengths. The support from collaboratives’ local partnerships was key to reaching their goals, highlighting the role community engagement and cross-sector collaboration plays in a large-scale initiative. Providing optimal support to each participating collaborative given the varying stages of readiness for program implementation was a noted challenge with such a large-scale initiative. Conclusion The OSOW program provides an example of a CIM initiative that engages program participants and stakeholders as meaningful participants in the evaluation. Results from the evaluation provide insights into the benefits and challenges of cross-sector engagement of a CIM initiative that other evaluators may apply to their work.