Using big data for higher education evaluation and assessment

Abstract Information: The big data phenomena provides a challenge for stakeholders to process data and transform findings into credible evidence. What are tools stakeholders can use to process large amounts of data for national program evaluation? The project focuses on employing machine learning techniques to further develop the range of applications for higher education assessment and evaluation. While the technical aspects of machine learning continue to advance, the application of machine learning and use of machine learning model findings remain limited. For example, the Allegany County Fragile Families Challenge implemented machine learning using administrative data to predict social and educational outcomes (Salganik et al., 2019). However, the application of machine learning methods in the higher education field remains limited despite the large volume, variety, and velocity at which administrative data grows and changes. Further, the growing need for evidence across public policy and local program decision making engages evaluators to produce credible information. The Institute of Education Sciences published the What Works Clearinghouse (WWC), which provides formal examples of credible evidence. However, the methods for building credible evidence in the education policy context are limited to randomized control trials, quasi experimental design, and comparative case studies. The limitations might leverage a variety of data sources but have limited generalizability nationally leaving policy makers limited information for federal policymaking. While solutions have been implemented into education policymaking by requiring reference to the WWC for grantmaking, methods for program evaluation provide limited flexibility for program innovation. Since WWC requires programs to build credible evidence using RCTs or quasi experimental designs, limited services or interventions might take place and limit policymakers knowledge about what might work including when and where. Local contexts provide rich knowledge about how programs are implemented and the local outcomes that lead to long term policy goals. By incorporating machine learning to process big administrative data, the process for evaluating program outcomes and creating credible evidence for education policy may become easier. The current project covers how statistical and machine learning techniques can be used to predict participant outcomes relative to available features. The primary example employs machine learning models for participant matching and comparison group design in a causal inference setting. The implications for machine learning and evaluation in higher education include 1) predicting institutional level trends, and 2) the potential for process mapping and predicting student enrollment and success in higher education institutions.