TY - JOUR
T1 - Applying Bayesian Modeling and Receiver Operating Characteristic Methodologies for Test Utility Analysis
AU - Wang, Qiu
AU - Diemer, Matthew A.
AU - Maier, Kimberly S.
N1 - Funding Information:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is based on work supported by a Kuder, Inc. research grant awarded to the first and the second authors.
Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.
PY - 2013/4
Y1 - 2013/4
N2 - This study integrated Bayesian hierarchical modeling and receiver operating characteristic analysis (BROCA) to evaluate how interest strength (IS) and interest differentiation (ID) predicted low-socioeconomic status (SES) youth's interest-major congruence (IMC). Using large-scale Kuder Career Search online-assessment data, this study fit three models, the one-level BROCA, the two-level BROCA, and the ordinal Probit BROCA, to examine the moderating effects of gender and race/ethnicity. Both IS and ID displayed race/ethnicity differences in predicting low-SES females' IMC. Gender difference was found only on IS in predicting low-SES youth's IMC. Results suggested that low-SES White males and low-SES minority females may need help the most to develop stronger career interests and to differentiate their interests. This study illustrated that BROCA can be a powerful tool for test evaluation and utility analysis in the field because of its capacity of analyzing continuous, nominal, and ordinal data; its graphical nature of result presentation; multiple statistical test options; and its little requirement of Level 2 sample sizes.
AB - This study integrated Bayesian hierarchical modeling and receiver operating characteristic analysis (BROCA) to evaluate how interest strength (IS) and interest differentiation (ID) predicted low-socioeconomic status (SES) youth's interest-major congruence (IMC). Using large-scale Kuder Career Search online-assessment data, this study fit three models, the one-level BROCA, the two-level BROCA, and the ordinal Probit BROCA, to examine the moderating effects of gender and race/ethnicity. Both IS and ID displayed race/ethnicity differences in predicting low-SES females' IMC. Gender difference was found only on IS in predicting low-SES youth's IMC. Results suggested that low-SES White males and low-SES minority females may need help the most to develop stronger career interests and to differentiate their interests. This study illustrated that BROCA can be a powerful tool for test evaluation and utility analysis in the field because of its capacity of analyzing continuous, nominal, and ordinal data; its graphical nature of result presentation; multiple statistical test options; and its little requirement of Level 2 sample sizes.
KW - Bayesian ROC analysis
KW - classification analysis
KW - utility analysis
UR - http://www.scopus.com/inward/record.url?scp=84874542965&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84874542965&partnerID=8YFLogxK
U2 - 10.1177/0013164412455027
DO - 10.1177/0013164412455027
M3 - Article
AN - SCOPUS:84874542965
SN - 0013-1644
VL - 73
SP - 275
EP - 292
JO - Educational and Psychological Measurement
JF - Educational and Psychological Measurement
IS - 2
ER -