123 - Examining the validity of retrospective pre-post assessments for measuring entrepreneurial mindset

Abstract Information: This poster focuses on a measure developed for a National Science Foundation Advanced Technological Education (NSF ATE) funded project focused on increasing students’ entrepreneurial mindset through a product design program. The external evaluation team and project team collaborated to develop a pre-post assessment to measure students’ entrepreneurial mindset. They used an existing framework of entrepreneurial mindset to organize the constructs of interest for the assessment and used various subscales from existing, validated measures to capture the constructs identified in the framework. They also developed their own scale to measure the product design and business skills they anticipated the project would impact. Evaluators administered the pre-post assessment to and conducted focus groups with the first cohort of students who participated in the 2023 program.

In both the pre- and post-assessment, students rated themselves very highly regarding teamwork and collaboration. However, during focus groups, students shared their greatest learnings were around working and collaborating with others. These findings led evaluators to consider the utility of alternative methods to examine change in the outcome of interest, such as retrospective pre-posttests. This method of data collection involves participants providing their “pre” and “post” ratings at the same time, addressing the potential response shift bias observed from students participating in this program.

For the second cohort, evaluators administered the pre-assessment before the start of the 2024 program. They will administer a retrospective pre-post assessment at the conclusion of the program, which will ask students to provide ratings for each question before and after their participation in the program. Evaluators will conduct follow up focus groups with students to better understand the changes in entrepreneurial mindset they experienced throughout the program and whether students feel they experienced a response shift bias. Specifically, evaluators will provide students with their assessment ratings and ask them to think about whether the skills and knowledge they developed throughout the program “shifted” their ratings from the “true” pre to the retrospective pre.

This poster will include results that compare the three sets of ratings: (1) the “true” pre-assessment ratings, (2) the retrospective pre-assessment ratings, and (3) the “true” post-assessment ratings. Evaluators will report the reliability statistics for the three sets of ratings. The poster will also include the focus group findings. Evaluators will triangulate the quantitative and qualitative findings to determine whether a retrospective pre-post assessment is more appropriate than a true pre-post assessment for measuring change in entrepreneurial mindset and will identify implications for other related constructs and outcomes.