160 - A Postmortem Analysis of an Experiment in Higher Education

Abstract Information: In this session, participants will consider the successes and shortcomings of an experiment conducted in higher education. It is true that experiments can seem daunting in practical settings, but this case study will describe how an alternative-treatments pre-post experimental design isolated the impacts of a learning intervention that took place in three different post-secondary courses. What design choices were critical? What practical constraints challenged a rigorous design? Was the experiment successful at revealing the effectiveness of the intervention? We will discuss these questions from a program evaluator’s perspective and address creative solutions to common challenges. In this case, an experiment was part of a mixed methods program evaluation to understand whether a campuswide initiative was improving the student experience. There were successes and shortcomings related to assessment, statistical testing, and data visualizations that provide guidance for the future.