Exploring Discrepancies in Pre-Post and Retrospective Survey Responses

Abstract Information: This study explores the nuances and distinctions between pre-post and retrospective measurements, investigating their implications, particularly in the context of assessing changes in responses before and after interventions. Pre-post measurements are often subject to ceiling effects, which occur when participants’ pre scores are too high to demonstrate significant gains in their post scores. Our research aims to discern whether variations in these measurements differ by social identities or the constructs being assessed. The questions addressed include: Are there differences between pre-post and retrospective answers? Do these differences depend on social identities? Do these differences depend on the construct? Amplifying and empowering voices requires accurate assessment of how initiatives affect them. Using survey data (n = 116) from participants in an intervention seeking to increase the sense of belongingness and scientific identity among underrepresented students by creating small communities, we analyzed changes in pre-post and retrospective answers across racial/ethnic groups and constructs: sense of belonging and scientific identity. Participants included: 57% women, 43% men, 64% first generation college students, 5% Asian, 7% Black, 33% Hispanic/Latine, 6% Native American, 41% White, 7% Multiracial, 2% Other Results revealed significantly higher changes in belonging and scientific identity when measured retrospectively compared to pre-post. For instance, retrospective measurement showed a mean change of 0.61 in belonging, significantly different from the 0.30 change observed with pre-post (t(116) = 3.06, p = .002). Similarly, for scientific identity, retrospective measurement had a mean change of 0.29, significantly different from the 0.04 observed with the pre-post (t(116) = 3.65, p < .001). Both methods of measurement found significant increases in belongingness and scientific identity before and after the intervention, but the magnitude differed with the retrospective assessments showing larger mean differences in both cases. These findings suggest that retrospective measurement may be more sensitive in detecting changes in these constructs. Additional findings suggest variations in the change of belonging and scientific identity across racial/ethnic groups, with retrospective measurements often showing higher mean changes compared to pre-post measurements. Using pre-post measurement, Asian American and Black students reported higher scientific identity scores and Native American students reported higher sense of belonging scores before the intervention, but all groups reported higher scores in both constructs after the intervention when using retrospective measurement. Although susceptible to response-shift bias (Howard, 1980), pre-post measurement is suitable when assessing the short-term effects of interventions, such as measuring factual knowledge or skills at two specific times. Retrospective measurement, a cost-effective and efficient alternative to longitudinal studies, is suitable for assessing liberatory outcomes (Garcia, 2022) or past experiences and behaviors, especially when direct measurement at the time of occurrence is impractical. Evaluators should carefully weigh research objectives, time frames, and potential biases when selecting measurement methods. Overall, our findings underscore the importance of methodological considerations in evaluating interventions, particularly regarding the choice between pre-post and retrospective measurements.