11 - Measuring Collaboration: Adapting Open-source Collaboration Assessment Tools for Virtual Dissemination

Abstract Information: Assessing collaboration in evaluation is increasingly common with the development of qualitative and quantitative tools such as the Levels of Collaboration Scale (Frey et al., 2004) and the Wilder Factors of Collaboration Inventory (Matteschich 2001). However, these tools are traditionally implemented “on paper” and in person, which may limit the scalability and data management of such tools compared with surveys distributed and completed through online platforms. As part of our evaluation of Early Learning Collaboratives (ELCs) in Idaho, we conducted a novel study of collaboration using these two open-source, freely available tools. The Levels of Collaborations Scale is a survey tool that provides a quantitative measure of the strength of collaborative relationships along a five-stage continuum from networking to full collaboration (Frey et al., 2004). Noonan and colleagues (2014) developed an approach to using the Levels of Collaboration Scale by combining social network analysis approach with a Utilization-Focused Evaluation. We adapted this approach for our multi-site evaluation using an online survey. In our evaluation design, we also included the Wilder Factors of Collaboration Inventory, which is a survey tool that qualitatively assesses 22 factors influencing collaborative efforts. Our purpose was to collect baseline data for assessing changes to the organization and collaborative strengths of ELCs in Idaho over a three-year period. These tools were selected because they are free and accessible for program implementation partners to (re)use even after the evaluation is complete. We recruited partners from 17 ELCs across the state of Idaho to participate in the study and adapted the survey instruments to online versions distributed through RedCap. This presentation will discuss the practical challenges encountered and strategies for survey design, recruitment, and data analysis in adapting these collaboration assessment tools for digital distribution. For example, we will discuss how we developed online surveys with measures designed for in-person utilization, facilitated online responses from partners, and made analytical decisions about missing data.

References: Mattessich, P. W., Murray-Close, M., & Monsey, B. R. (2001). Collaboration: What makes it work. 2nd ed., St. Paul, MN: Amherst Wilder Foundation. Frey, B. B., Lohmeier, J. H., Lee, S. W., & Tollefson, N. (2006). Measuring collaboration among grant partners. American Journal of Evaluation, 27(3), 383. Noonan, P., Gaumer Erickson, A., McCall, Z., Frey, B. B., & Zheng, C. (2014). Evaluating change in interagency collaboration of a state-level interagency education team: A social network approach within a utilization-focused framework. Educational Assessment, Evaluation and Accountability, 26(3), 301–316. https://doi.org/10.1007/s11092-014-9193-2