Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

Jeffrey J. Starns; Andrea M. Cataldo; Caren M. Rotello; Jeffrey Annis; Andrew Aschenbrenner; Arndt Bröder; Gregory Cox; Amy Criss; Ryan A. Curl; Ian G. Dobbins; John Dunn; Tasnuva Enam; Nathan J. Evans; Simon Farrell; Scott H. Fraundorf; Scott D. Gronlund; Andrew Heathcote; Daniel W. Heck; Jason L. Hicks; Mark J. Huff; David Kellen; Kylie N. Key; Asli Kilic; Karl Christoph Klauer; Kyle R. Kraemer; Fábio P. Leite; Marianne E. Lloyd; Simone Malejka; Alice Mason; Ryan M. McAdoo; Ian M. McDonough; Robert B. Michael; Laura Mickes; Eda Mizrak; David P. Morgan; Shane T. Mueller; Adam Osth; Angus Reynolds; Travis M. Seale-Carlisle; Henrik Singmann; Jennifer F. Sloane; Andrew M. Smith; Gabriel Tillman; Don van Ravenzwaaij; Christoph T. Weidemann; Gary L. Wells; Corey N. White; Jack Wilson

doi:10.1177/2515245919869583

Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

Jeffrey J. Starns, Andrea M. Cataldo, Caren M. Rotello, Jeffrey Annis, Andrew Aschenbrenner, Arndt Bröder, Gregory Cox, Amy Criss, Ryan A. Curl, Ian G. Dobbins, John Dunn, Tasnuva Enam, Nathan J. Evans, Simon Farrell, Scott H. Fraundorf, Scott D. Gronlund, Andrew Heathcote, Daniel W. Heck, Jason L. Hicks, Mark J. HuffDavid Kellen, Kylie N. Key, Asli Kilic, Karl Christoph Klauer, Kyle R. Kraemer, Fábio P. Leite, Marianne E. Lloyd, Simone Malejka, Alice Mason, Ryan M. McAdoo, Ian M. McDonough, Robert B. Michael, Laura Mickes, Eda Mizrak, David P. Morgan, Shane T. Mueller, Adam Osth, Angus Reynolds, Travis M. Seale-Carlisle, Henrik Singmann, Jennifer F. Sloane, Andrew M. Smith, Gabriel Tillman, Don van Ravenzwaaij, Christoph T. Weidemann, Gary L. Wells, Corey N. White, Jack Wilson

Research output: Contribution to journal › Article › peer-review

15 Scopus citations

Abstract

Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.

Original language	English (US)
Pages (from-to)	335-349
Number of pages	15
Journal	Advances in Methods and Practices in Psychological Science
Volume	2
Issue number	4
DOIs	https://doi.org/10.1177/2515245919869583
State	Published - Dec 2019

Keywords

Bayesian methods
blinded inference
bootstrap
memory
metascience
modeling
open data

ASJC Scopus subject areas

General Psychology

Access to Document

10.1177/2515245919869583

Cite this

Starns, J. J., Cataldo, A. M., Rotello, C. M., Annis, J., Aschenbrenner, A., Bröder, A., Cox, G., Criss, A., Curl, R. A., Dobbins, I. G., Dunn, J., Enam, T., Evans, N. J., Farrell, S., Fraundorf, S. H., Gronlund, S. D., Heathcote, A., Heck, D. W., Hicks, J. L., ... Wilson, J. (2019). Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis. Advances in Methods and Practices in Psychological Science, 2(4), 335-349. https://doi.org/10.1177/2515245919869583

Starns, JJ, Cataldo, AM, Rotello, CM, Annis, J, Aschenbrenner, A, Bröder, A, Cox, G, Criss, A, Curl, RA, Dobbins, IG, Dunn, J, Enam, T, Evans, NJ, Farrell, S, Fraundorf, SH, Gronlund, SD, Heathcote, A, Heck, DW, Hicks, JL, Huff, MJ, Kellen, D, Key, KN, Kilic, A, Klauer, KC, Kraemer, KR, Leite, FP, Lloyd, ME, Malejka, S, Mason, A, McAdoo, RM, McDonough, IM, Michael, RB, Mickes, L, Mizrak, E, Morgan, DP, Mueller, ST, Osth, A, Reynolds, A, Seale-Carlisle, TM, Singmann, H, Sloane, JF, Smith, AM, Tillman, G, van Ravenzwaaij, D, Weidemann, CT, Wells, GL, White, CN & Wilson, J 2019, 'Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis', Advances in Methods and Practices in Psychological Science, vol. 2, no. 4, pp. 335-349. https://doi.org/10.1177/2515245919869583

@article{e4923361d19d45f2acff3fa0273d64b2,

title = "Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis",

abstract = "Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts{\textquoteright} judgments on the very same data, hinting at a possible inference crisis.",

keywords = "Bayesian methods, blinded inference, bootstrap, memory, metascience, modeling, open data",

author = "Starns, {Jeffrey J.} and Cataldo, {Andrea M.} and Rotello, {Caren M.} and Jeffrey Annis and Andrew Aschenbrenner and Arndt Br{\"o}der and Gregory Cox and Amy Criss and Curl, {Ryan A.} and Dobbins, {Ian G.} and John Dunn and Tasnuva Enam and Evans, {Nathan J.} and Simon Farrell and Fraundorf, {Scott H.} and Gronlund, {Scott D.} and Andrew Heathcote and Heck, {Daniel W.} and Hicks, {Jason L.} and Huff, {Mark J.} and David Kellen and Key, {Kylie N.} and Asli Kilic and Klauer, {Karl Christoph} and Kraemer, {Kyle R.} and Leite, {F{\'a}bio P.} and Lloyd, {Marianne E.} and Simone Malejka and Alice Mason and McAdoo, {Ryan M.} and McDonough, {Ian M.} and Michael, {Robert B.} and Laura Mickes and Eda Mizrak and Morgan, {David P.} and Mueller, {Shane T.} and Adam Osth and Angus Reynolds and Seale-Carlisle, {Travis M.} and Henrik Singmann and Sloane, {Jennifer F.} and Smith, {Andrew M.} and Gabriel Tillman and {van Ravenzwaaij}, Don and Weidemann, {Christoph T.} and Wells, {Gary L.} and White, {Corey N.} and Jack Wilson",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2019.",

year = "2019",

month = dec,

doi = "10.1177/2515245919869583",

language = "English (US)",

volume = "2",

pages = "335--349",

journal = "Advances in Methods and Practices in Psychological Science",

issn = "2515-2459",

publisher = "SAGE Publications Inc.",

number = "4",

}

TY - JOUR

T1 - Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

AU - Starns, Jeffrey J.

AU - Cataldo, Andrea M.

AU - Rotello, Caren M.

AU - Annis, Jeffrey

AU - Aschenbrenner, Andrew

AU - Bröder, Arndt

AU - Cox, Gregory

AU - Criss, Amy

AU - Curl, Ryan A.

AU - Dobbins, Ian G.

AU - Dunn, John

AU - Enam, Tasnuva

AU - Evans, Nathan J.

AU - Farrell, Simon

AU - Fraundorf, Scott H.

AU - Gronlund, Scott D.

AU - Heathcote, Andrew

AU - Heck, Daniel W.

AU - Hicks, Jason L.

AU - Huff, Mark J.

AU - Kellen, David

AU - Key, Kylie N.

AU - Kilic, Asli

AU - Klauer, Karl Christoph

AU - Kraemer, Kyle R.

AU - Leite, Fábio P.

AU - Lloyd, Marianne E.

AU - Malejka, Simone

AU - Mason, Alice

AU - McAdoo, Ryan M.

AU - McDonough, Ian M.

AU - Michael, Robert B.

AU - Mickes, Laura

AU - Mizrak, Eda

AU - Morgan, David P.

AU - Mueller, Shane T.

AU - Osth, Adam

AU - Reynolds, Angus

AU - Seale-Carlisle, Travis M.

AU - Singmann, Henrik

AU - Sloane, Jennifer F.

AU - Smith, Andrew M.

AU - Tillman, Gabriel

AU - van Ravenzwaaij, Don

AU - Weidemann, Christoph T.

AU - Wells, Gary L.

AU - White, Corey N.

AU - Wilson, Jack

PY - 2019/12

Y1 - 2019/12

N2 - Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.

AB - Scientific advances across a range of disciplines hinge on the ability to make inferences about unobservable theoretical entities on the basis of empirical data patterns. Accurate inferences rely on both discovering valid, replicable data patterns and accurately interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded-inference paradigm for assessing the quality of theoretical inferences from data. Our results reveal substantial variability in experts’ judgments on the very same data, hinting at a possible inference crisis.

KW - Bayesian methods

KW - blinded inference

KW - bootstrap

KW - memory

KW - metascience

KW - modeling

KW - open data

UR - http://www.scopus.com/inward/record.url?scp=85103030306&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85103030306&partnerID=8YFLogxK

U2 - 10.1177/2515245919869583

DO - 10.1177/2515245919869583

M3 - Article

AN - SCOPUS:85103030306

SN - 2515-2459

VL - 2

SP - 335

EP - 349

JO - Advances in Methods and Practices in Psychological Science

JF - Advances in Methods and Practices in Psychological Science

IS - 4

ER -

Assessing Theoretical Conclusions With Blinded Inference to Investigate a Potential Inference Crisis

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this