Methodological adaptations to concept mapping for improved planning and evaluation

Abstract Information:

Background: Concept mapping, also called Group Concept Mapping, is a mixed-method approach that has been widely used in program planning and evaluation to understand group thinking, generate conceptual frameworks, and support decision making. Initially, participants are prompted to provide elements of reflection on a specific issue, which they then sort based on conceptual similarities. Multivariate analysis, using nonmetric multidimensional scaling (nMDS) and clustering analysis, is then typically used to identify overarching themes. The outcome is represented visually as a two-dimensional map (the concept map) where the clustered ideas reflect the emerging themes and collective understanding of the issue. This map serves as a potent tool to encourage discussion and reflexivity and to inform future planning and evaluation endeavors. Although this method has been deployed extensively for decades, there has been limited discourse concerning the statistical foundations it is built upon. One concern is that traditionally, the distances computed from the statement sortings are first projected into lower-dimensional spaces using nMDS before subjecting them to clustering analysis. This practice has been questioned because running cluster analysis after nMDS might adversely affect the validity of the resulting concept map.

Objectives: We wanted to test if conducting the clustering analysis directly on the original sorting distances before projecting the data into lower-dimensional spaces yielded better performance than doing the clustering after the dimensionality reduction. We also test the performance of other dimensionality reduction approaches beyond nMDS.

Method: We conducted a simulation study comparing the performance of both methods (cluster analysis before vs. after dimensionality reduction) using varying experimental parameters regarding number of items (from 40 to 120), number of clusters (4 to 12) and the level of overlapping when computing the simulated distance matrices. We then evaluated which methods better identified the original clusters using adjusted mutual information scores. Then, we compared three dimensionality reduction algorithms – nMDS, isomap and spectral, to see which provided a better “visualization” of the clusters. This was assessed by looking at the degree of overlapping between enclosing polygons (i.e., convex hull) in 2D space using data from existing projects.

Results: The outcomes of our simulation study appear to corroborate the assertion that our alternative approach yields a more precise delineation of the underlying clustering structure. Using simulated distance matrices with 60 items structured in 6 clusters, the adjusted mutual information score was, on average, 0.13 points higher (95% CI = +/-0.02) when clustering was performed on the raw distance matrices rather than on the nMDS 2D projections. This improvement in accuracy becomes even stronger as the number of items or clusters increases, and particularly when the stress index is elevated, which is characteristic of concept mapping data. Furthermore, compared to nMDS, isomap and spectral algorithms provide a clearer representation of the clustered ideas when the clustering analysis is applied on the original distance matrix.

Conclusion: The proposed modifications of the way concept mapping data is analyzed leads to more valid and improved visualization of concept maps, thereby providing better guidance for planning and evaluation processes.