268 - Examining Method Effects of Keying and Wording in Psychometric Instruments: A Quantitative Method in Evaluation

Abstract Information: This study presents an innovative approach when applying quantitative methods to conduct the program evaluation. By employing a quantitative approach, the keying method, wording method, and construct validity in two psychometric assessments were examined. Those two self-report psychometric assessments were the Effortful Control assessment (Ellis & Rothbart, 2001) and the Grit assessment (Duckworth & Quinn, 2009). Moreover, this study utilized structural equation modeling (SEM) to analyze 2,104 students’ responses and to assess the construct of keying and wording. 11 hypothetical models were investigated and evaluated. The reliability of each construct was analyzed using several omega coefficients. Simultaneously, construct validity was discussed through model performance and parameter estimates. The main problem is that the method effects of wording and keying can contaminate substantive results (Carmines & Zeller, 1979; Chen, 2017; Coleman, 2013). There is a gap in the literature that addresses the keying method, which are Negative Keying (NK) and Positive Keying (PK). In particular, Negative Keying Positive Wording (NKPW), describes an item that has an opposite direction of the semantic meaning from the trait being measured but without any negation markers in the item stem. Positive Keying Positive Wording (PKPW) represents an item that has the same direction of the semantic meaning from a trait being measured and does not use any negation markers in the item stem. For example, suppose a five-point Likert scale was created to measure students' school engagement. The lowest response score of 1 denotes the least engagement in school activities and the highest score of 5 reflects the most interest in school events. In this case, an NKPW item would be “I want to stay alone during school events”. A PKPW item would be "I like school activities". Accordingly, the main purpose of this research was to examine two keying methods, NK and PK, through several SEMs. In addition, insufficient research assessed the joint effects of the keying method and the wording method, which are Negative Keying Negative Wording (NKNW) and Positive Keying Positive Wording (PKPW). Thus, the second purpose of this study was to examine the method effects of keying and wording, which are NKNW and PKPW. Findings revealed that the keying method was detected, although the structure of keying displayed inconsistent probability of variance associated with the reliability across different assessments. Additionally, method effects of keying and wording were identified, and problematic item questions were further examined. This study therefore addressed research gaps regarding the effects of keying and wording in psychometric instruments and contributed to understanding their impact on construct validity. The findings draw attention to the keying and wording methods in self-report psychometric instruments when applying the quantitative method to conduct program evaluation. This also impacted the accuracy of assessing the construct of interest and the further parameter estimates. The presenter will share some lessons learned from the analysis, and some key insights that evaluators can use when creating or reviewing surveys for their evaluations. Based on these findings, suggestions for future quantitative methods in program evaluation were discussed and summarized.