Replication and Reproducibility

Uljana Feest (Leibniz Universität Hannover)
Current debates about the replication crisis in psychology take it for granted that direct replication is valuable and focus their attention on questionable research practices in regard to statistical analyses. This paper takes a broader look at the notion of replication as such. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual and material questions in research, and I argue that replication is not as central to psychological research as it is sometimes taken to be.

Rafael Ventura (Duke University)
Science is a social enterprise. To make progress, scientists must assume that the results of others are at least in principle reproducible. But empirical studies show that researchers across different disciplines often fail to reproduce results from previous experiments. To explain this lack of reproducibility, two main hypotheses have been proposed: (1) publication bias is the main cause of low reproducibility; (2) lax standard of statistical significance is the main cause of low reproducibility. Here, I present a model to adjudicate between these two hypotheses. Model results suggest that publication bias may play a more important role than current levels of statistical significance in promoting low reproducibility.

Justin Bruner (Australian National University), Bennett Holman (Underwood International College, Yonsei University)
Concern over the reproducibility of experimental work in the social sciences has motivated some to re-examine the extent to which science can be said to be self-correcting. We consider a recent argument put forth by Romero (2016) that science is unlikely to self-correct because of its social structure and the norms that govern publication practices. We contend this understanding of scientific self-correction is misguided and argue that self-correction is possible but requires both a norm of truth seeking and a commitment to the development of new inferential techniques and data aggregation procedures.

Deborah Mayo (Virginia Tech)
This paper discusses the 2011-2015 Reproducibilty Project, an attempt to replicate published statistically significant results in psychology. We set out key elements of significance tests, often misunderstood. While intended to bound the probabilities of erroneous interpretations of data, this error control is nullified by cherry-picking, multiple testing, and other biasing selection effects. However, the reason to question the resulting inference is not a matter of poor long-run error rates, but rather that it has not been well-tested by these data. This provides a rationale never made clear by significance testers as to the inferential relevance of error probabilities.