TY - JOUR
T1 - Validating automated integrative complexity
T2 - Natural language processing and the Donald Trump test
AU - Conway, Lucian Gideon
AU - Conway, Kathrene R.
AU - Houck, Shannon C.
N1 - Publisher Copyright:
© 2020, PsychOpen. All rights reserved.
PY - 2020
Y1 - 2020
N2 - Computer algorithms that analyze language (natural language processing systems) have seen a great increase in usage recently. While use of these systems to score key constructs in social and political psychology has many advantages, it is also dangerous if we do not fully evaluate the validity of these systems. In the present article, we evaluate a natural language processing system for one particular construct that has implications for solving key societal issues: Integrative complexity. We first review the growing body of evidence for the validity of the Automated Integrative Complexity (AutoIC) method for computer-scoring integrative complexity. We then provide five new validity tests: AutoIC successfully distinguished fourteen classic philosophic works from a large sample of both lay populations and political leaders (Test 1) and further distinguished classic philosophic works from the rhetoric of Donald Trump at higher rates than an alternative system (Test 2). Additionally, AutoIC successfully replicated key findings from the hand-scored IC literature on smoking cessation (Test 3), U.S. Presidents’ State of the Union Speeches (Test 4), and the ideology-complexity relationship (Test 5). Taken in total, this large body of evidence not only suggests that AutoIC is a valid system for scoring integrative complexity, but it also reveals important theory-building insights into key issues at the intersection of social and political psychology (health, leadership, and ideology). We close by discussing the broader contributions of the present validity tests to our understanding of issues vital to natural language processing.
AB - Computer algorithms that analyze language (natural language processing systems) have seen a great increase in usage recently. While use of these systems to score key constructs in social and political psychology has many advantages, it is also dangerous if we do not fully evaluate the validity of these systems. In the present article, we evaluate a natural language processing system for one particular construct that has implications for solving key societal issues: Integrative complexity. We first review the growing body of evidence for the validity of the Automated Integrative Complexity (AutoIC) method for computer-scoring integrative complexity. We then provide five new validity tests: AutoIC successfully distinguished fourteen classic philosophic works from a large sample of both lay populations and political leaders (Test 1) and further distinguished classic philosophic works from the rhetoric of Donald Trump at higher rates than an alternative system (Test 2). Additionally, AutoIC successfully replicated key findings from the hand-scored IC literature on smoking cessation (Test 3), U.S. Presidents’ State of the Union Speeches (Test 4), and the ideology-complexity relationship (Test 5). Taken in total, this large body of evidence not only suggests that AutoIC is a valid system for scoring integrative complexity, but it also reveals important theory-building insights into key issues at the intersection of social and political psychology (health, leadership, and ideology). We close by discussing the broader contributions of the present validity tests to our understanding of issues vital to natural language processing.
KW - AutoIC
KW - Automated Integrative Complexity
KW - Integrative complexity
KW - Natural language processing
UR - http://www.scopus.com/inward/record.url?scp=85090543555&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85090543555&partnerID=8YFLogxK
U2 - 10.5964/jspp.v8i2.1307
DO - 10.5964/jspp.v8i2.1307
M3 - Article
AN - SCOPUS:85090543555
SN - 2195-3325
VL - 8
SP - 504
EP - 524
JO - Journal of Social and Political Psychology
JF - Journal of Social and Political Psychology
IS - 2
ER -