Estimating Mediation Effects in Experimental Evaluations with Partially Nested Data

Abstract Information: With increasing emphasis on capturing a more comprehensive understanding of intervention effects, analyses of experimental evaluations should reflect the complex relationships under investigation. Structural equation modeling is an effective analytic approach in these settings but requires large sample sizes (Li & Beretvas, 2013) that can be impractical (e.g., Schochet, 2011). Additionally, interventions often involve group-administered or shared facilitator interventions that produce disparate grouping or nesting structures across treatment conditions. A structure-after-measurement approach using Croon’s corrections (SAM-Croon’s; Croon, 2002; Rosseel & Loh, nd) has been effectively applied with limited sample sizes and complex SEMs (e.g., Devlieger & Rosseel, 2017; Kelcey et al., 2020) but it has not been comprehensively evaluated with partially nested data common in experimental evaluations (e.g., Bauer et al., 2008; Lohr et al., 2014). Based on encouraging results from recent literature (Authors Masked; Devlieger & Rosseel, 2019; Kelcey, 2019; Kelcey, et al., 2020), the purpose of this study was to extend and evaluate SAM-Croon’s to estimate mediation effects in structural equation models (SEM) with partially nested data. Specifically, we develop the extensions and use simulation studies to compare mediation effect estimates for multilevel SEMs using SAM-Croon’s, ML, and a SAM approach with uncorrected factor scores (SAM-FS) when data are partially nested. Our Monte Carlo simulation studies vary sample sizes, mediation effects, variance decomposition of latent variables, and measurement model factors (e.g., indicator weights and decomposition). For didactic purposes and space constraints, we limit this proposal to SAM-Croon’s estimation with a 2/1 partially nested design and a single mediation effect from an individual-level mediator. Two/one partial nesting occurs when one study arm contains two-levels (i.e., students nested within classroom intervention groups) and the other study arm is represented in a single-level (i.e., waitlisted students in a control group). We draw on the multiple-arm multilevel structural equation modeling framework for partially nested data (Lachowicz, et al. 2015; Sterba et al., 2014) and use two analytic models that reflect the differing study arms (i.e., treatment and control groups). Our model captures the indirect effect of the treatment on the outcome through a mediator. Observed variables at the individual-level are decomposed into a latent between and within cluster component. We include both individual- and cluster-level covariates. Typical maximum likelihood estimation (ML) consistently provided results while SAM-FS and SAM-Croon’s did fail to provide results in a small portion of cases when cluster sample sizes were 15 or less. The performance of ML seems advantageous until bias and efficiency are considered. Convergence issues for SAM-FS and SAM-Croon’s did quickly dissipate as cluster sample sizes met and exceeded 30. ML failed to produce accurate or efficient mediation effect estimates when cluster sample sizes were less than 15. SAM-FS demonstrated its well documented bias across conditions due to its disregard of latent variable unreliability. Overall, all estimators performed well as cluster sample sizes met and exceeded 30. These limited results suggest Croon’s is an effective estimation approach for mediation effects in SEMs with partially nested data.