Improved Statistical Reporting to Stakeholders: Effect Sizes, Confidence Intervals and the Dichotomous Nature of p Values

Abstract Information: This presentation will provide an introduction to enlightened statistical reporting, in-line with the recent call by Amrhein, et. al. (2019) to abandon p values as a dichotomous criteria. Calls for the concept of statistical “significance” to be abandoned are becoming louder and louder; 800 signatories supported Amrhein et. al.’s commentary within its first week of publication and this commentary was followed by a special issue of The American Statistician titled “Statistical Inference in the 21st Century: A World Beyond “p < .05” (2019).

This session will set the stage with an overview of fundamental concepts, including essentials of null hypothesis testing, alpha, effect sizes and confidence intervals. These concepts will be used to show how we, as evaluators, can support our stakeholder’s understanding of p values in the move from dichotomous statements to descriptive continuous values (Wasserstein et. al., 2019). Practitioners will be better positioned to convey statistical findings beyond pass/fail significance statements. Using test data, and concrete case examples from recent evaluations, the presentation will highlight how significance statements leave out valuable contextual information, and how this can be improved. Further, examples will show how despite the same statistical p value, differing means and standard deviations can produce widely different effect sizes across analyses. An overview of interpreting effect sizes and confidence intervals to provide a more complete statistical picture to stakeholders will also be discussed, as will how new reporting norms can support evaluation findings from samples smaller than needed as determined by power analysis.

Not only will practitioners be given more tools to report and convey the meanings of statistical tests to stakeholders, they will also be empowered to remove the dividing line between p = .051 and p = .049 (Amrhein, et. al., 2019) and give more contextual information to stakeholders.

Audience members should expect to better understand the statistical concepts of null hypothesis, alpha, effect sizes and confidence intervals and how to use these concepts to inform stakeholder decision making beyond significance statements.