Isaac Davis (Carnegie Mellon University)
In epistemic reliability theory, verification and refutation are treated as success criteria for methods, rather than entailment relations between hypothesis and data (Kelly 2000). Utilizing formal learning theory, we can achieve deductive guarantees that a method will converge to the truth in the limit, even if deductive verification of hypotheses is impossible. This framework provides a formal characterization of underdetermination in scientific inquiry, and a basis for designing inference methods with deductively guaranteed truth-convergence and simplicity properties. In this paper, we investigate whether epistemic reliability criteria are applicable to economics, what methods satisfy this criteria, and how this can inform economic research decisions. We argue that, due to the nature of underdetermination inherent to economic research, the standard epistemic reliability framework does not apply to much of economics. However, in conjunction with a pragmatic research goal (e.g. predicting future phenomena, optimizing policy, or allocating resources), formal learning theory allows us to extract new, informative questions that admit a pragmatic reliability analysis, in the sense that we can obtain convergence-in-the-limit guarantees for achieving pragmatic goals. This method of extracting "coarser" questions from more "fine-grained" questions can be performed in a principled way, and the underlying procedure appears in different forms and different domains, such as the Adaptively Rational Learning framework in Wellen and Danks (2016) and the Causal Feature Learning framework in Chalupka et al (2017). We show how this criterion can be defined and applied in economics, and how it relates to the epistemic equivalent. Finally, we demonstrate that, in a special case where the pragmatic goal is prediction of future phenomena, the pragmatic and epistemic notions of reliability coincide in an important way. This equivalence provides some justification for the common view of economics as being primarily concerned with prediction of future phenomena, rather than discovering "true'' underlying causal mechanisms (e.g. Friedman 1953, McCloskey 1998).
