8 - Retrospective Post-then-Pre Questionnaire Design: Looking Back at What We Learned

Abstract Information: At the beginning of an educational experience, it can be difficult to assess what we know, what we don’t know, and what we “don’t know that we don’t know”, and what we “don’t know that we do know”. This is especially challenging when a pre-test contains terms that may be unfamiliar to participants or utilize technical language that will be further explained in the educational experience. Situations like this make assessment and evaluation challenging for both the participant and the evaluator. One methodological design choice that evaluators have is to use a retrospective post-then-pre questionnaire design that allows participants to share their perceived changes in knowledge, skills, and attitudes after completing the educational program. This allows participants to self-assess what they have learned after experiencing the program which allows them to acquire a common language and assess their progress from a specific frame of reference, for example, after learning key phrases and application of terminology. By providing the assessment after participants have access to a meaningful benchmark, participants can more accurately understand their own knowledge more clearly on novel topics. The strengths of the retrospective post-then-pre questionnaire design include reducing response shift bias, enhancing validity, reducing evaluation burden, and simplifying data handling. Administering the pre and post-tests after the educational experience creates a space where participants’ frames of reference are level set, which can help in reducing response shift bias and help evaluators unmask the true impact of programming. The validity of the instrument is also increased when the participants, program administrators, and evaluators are all using a shared language. Additionally, this methodology allows for simpler data collection practices while supporting both confidentiality and linkage of paired data. From an equity standpoint, there are clear benefits to the retrospective post-then-pre questionnaire design. With this design, the experiences of the participants are centered and participants are given the opportunity to more accurately report their knowledge before and after the program when they more fully understand the concepts that on which they are being assessed. Assessing at the end of the program also allows time for trust building, which is important in equity-deserving populations, which could reduce responses impacted by social desirability. Importantly, this design allows for the analysis of paired data without the need to link the paired data which creates space for a completely anonymous survey if that is desirable. Finally, the use of a single survey reduces the evaluation burden on both the administrators and the participants. Every methodological decision has trade-offs that must be considered and the retrospective post-then-pre questionnaire design is no different. Liabilities for this design include that the data is self-reported, participants’ abilities to recall prior status remains even though the questions are asked at the same time, and engaging in the survey in the same place where the training occurred can influence responses. Evaluators should also consider cultural context when choosing this or any design to ensure that it is the right match for the participants. Join us for an interactive session.