Improving empirical legal research (some comments on 'The Future of Empirical Legal Scholarship: Where Might We Go from Here?')
A guest post by Rory McFadden (UQ Law)
The fields of psychology, economics and biomedicine are all facing their own iteration of the same crisis: many empirical studies that have been peer-reviewed, published, and subsequently relied upon are now proving difficult or impossible to replicate (see Camerer et al, 2018). Where a study cannot be replicated – ie, where the same results cannot be obtained using the same methodology – its credibility is significantly diminished because it means either the replication effort or the original study is wrong.
The problem, it seems, is empiricism. Designing an empirical study and analysing its results entails a fair amount of discretion on the part of the researcher, which enables one of thousands of kinds of methodological error to creep in and skew the results (Altman, 1994). Investigations prompted by the crisis blame some combination of human and institutional biases, questionable research practises, and outright errors (Nelson et al, 2017).
Ignorance of the crisis has notable practical implications for law; for example, courts regularly rely on expert witness evidence without scrutiny of the quality of the study or studies relied on. However, legal scholars themselves also seem to be increasingly enthusiastic authors of empirical studies (Heise, 2011). Since researchers who are required to be trained in empirical methods can (and, perhaps surprisingly often, do) produce misleading results, it is not a far leap to suggest that legal scholars could do the same. This is especially true since such training in law is generally entirely voluntary, if it is available at all.
Herein lies the issue: few actors within the legal academic publication system have sufficient training to produce or verify high-quality empirical work. Alas, these concerns are not novel; academics have been worried about the accuracy of empirical conclusions in law for some time. Epstein and King (2011) presented concerns about the reliability of empirical legal research in their paper ‘The Rules of Inference’ almost two decades ago. More recently, Kathryn Zeiler’s 2016 article ‘The Future of Empirical Legal Scholarship: Where Might We Go from Here?’ discusses what she terms the ‘crisis of low quality’ within the legal academic publication system.
Zeiler points out that legal scholars are under no requirement to study empirical methods, and editorial boards, although keen to publish empirical studies, are under no obligation to employ an expert in reviewing them. In fact, many law journals in the US are student-run, and articles may never be reviewed by someone with sufficient expertise to spot methodological errors and limitations. Finally, there are seldom any reputational or other consequences for the publication of low-quality, flawed or erroneous empirical legal articles. In light of all this, there is simply no reason for law schools to pour funding into the resources necessary to improve. Hence, the system currently in place has the unfortunate effect of incentivising empiricism generally, while disincentivising the costly pursuit of high-quality empiricism. Lastly, all of this is seemingly rather high-stakes, since legal scholarship – especially with the apparent support of empirical evidence – can influence policy decisions and the law.
Commentators, concerned about the state of empirical legal scholarship, have been submitting proposals on how it might be improved for some time. One suggestion oft mentioned in the literature is that legal academics and the editors (often students) of legal journals should be required to undertake empirical training. However, Zeiler notes that existing empirical methods courses offered to legal academics may be insufficient, serving only to provide illusory credentials against which a flawed empirical analysis can be superficially propped up. Other suggestions submitted earnestly by concerned players over the years have failed to take off perhaps, Zeiler suggests, because they all fail to address the correct interests within the system: those of academics who submit low-quality empirical legal scholarship; of the editors who review and publish it; and, of the law schools who fund the whole endeavour.
Zeiler suggests that one interest common to both authors and editors within the legal field is reputation. Currently, low-quality empirical articles attract little public criticism unless their topic is particularly controversial. In addition, initiatives aimed at improving empirical scholarship generally (such as preregistration and data transparency) have insufficient reputational effect to incentivise participation by authors and journals alike. Zeiler suggests publication of expert reviews would incentivise authors to do better in order to protect their public image and assist readers who are unable to independently assess the quality of the work. This would also create a directory of errors that future authors can peruse with the aim to avoid. Furthermore, development of a grading system of law journals based on the quality of empirical publications (rather than impact and citation calculations) might encourage editors to be more critical to protect or increase their own prestige (See TOP).
Zeiler’s main argument is that initiatives aimed at improving the quality of empirical legal scholarship must capitalise on the interests of authors, editors and funding sources in order to be effective. Currently, the publication system in law is a breeding ground for low-quality empiricism; the actors within the system can only be expected to change if given good reason to. This is an important perspective on a continuing issue; however, it might be noted that similar calls to action – such as that of Epstein and King (2001) – have gone unanswered for years.
It is possible that a purely theoretical argument is simply not strong enough, and, ironically, might be strengthened with an empirical demonstration of many of the claims that are being made. A systematic review would need to initially look at what empirical legal scholarship actually is. No consistent definition emerges from the literature to date, although a good start is that of Epstein and King. They surmised empirical legal research as ‘learning about the world using quantitative data or qualitative information’. This implies that empiricism in legal scholarship may extend to, eg, simple case citations. Certainly, it is not common practise for legal academics to include a comprehensive list of search terms which brought them to the case being cited as authoritative law. If it was, then critiquing or replicating this methodology to ensure that cases weren’t being cherry-picked would be trivial. The next step would be to thoroughly examine – using a set of preregistered criteria – the replicability of a large sample of legal articles published in law journals.
Indeed, the same sort of systematic empirical review of replicability has helped spur change in other fields. It is unclear whether the same effect would be seen in legal academia. At least in terms of incentive, an empirical study demonstrating that legal scholarship is not immune from the same replication issues as in other fields may encourage legal academics to go out of their way to show that their study is in fact methodologically sound – unlike the rest.