It’s all relative: Explanationism and probabilistic evidence theory

Abstract

I provide a brief comment on Allen & Pardo’s “Relative plausibility and its critics”. I agree that their relative plausibility explanationism (RPE) provides an attractive positive description of fact finding in litigation. But unlike Allen & pardo, I see RPE as methodologically consistent with the leading conceptions of evidentiary probabilism, Bayesianism and likelihoodism. Thus, I argue, the best view of the relationship between the approaches is that RPE is a friendly amendment to–because it provides a positive foundation for–probabilism.

Keywords

evidence relative probability Bayesian likelihoodism

Ronald J. Allen and Michael S. Pardo (henceforth ‘AP’) offer an engaging and thought-provoking summary of the case for a paradigm shift toward what they call explanationism as the organising framework for evidence law theory (Allen and Pardo, 2019). AP’s article concentrates on the role of ‘relative plausibility’ in explanationism. I refer to AP’s proposed relative plausibility explanationism paradigm as ‘RPE’.

The Kuhnian incumbent paradigm is probabilism. Although the Bayesian approach to probabilism was first explicated precisely only in 1968, by John Kaplan, they write, ‘For literally hundreds of years, proof at trial was assumed to be probabilistic’ (Allen and Pardo, 2019: 5; Kaplan, 1968). According to AP, a core difference between RPE and probabilistic theory is that:

unlike the probability account, the explanatory account is inherently comparative—whether an explanation satisfies the standard depends on strength of the possible explanations supporting each side (and not only the party with the burden of proof). (Allen and Pardo, 2019: 15)

A primary objective of the present comment is to show, as recent work has demonstrated, that probabilism is capable of expression in comparative terms. Probabilistic approaches can be conveniently and powerfully expressed in terms of relative probabilities, raising the question of how different RPE and probabilism really are.

AP do recognise a ‘fundamental similarity’ in the paradigms: ‘that both the explanatory and probabilistic accounts focus on the same end: the likelihood of disputed facts’ (Allen and Pardo, 2019: 15). I agree with that sentiment. For precision, I distinguish two ‘probabilistic accounts’ to which scholars using conventional notions of probability might subscribe. The first, more familiar and popular notion, is the Bayesian approach that focuses on subjective posterior probabilities of states of the world, given the available evidence. The second account is likelihoodist. Unlike Bayesians, likelihoodists reject the use of subjective prior probabilities. That leaves likelihoodists focusing only on comparisons of the likelihood of the observed evidence under alternative possible states of the world.¹

I develop the distinction between these approaches in the first section below. As I demonstrate in the second section, both approaches can be viewed comparatively in a way that’s consistent with AP’s RPE framework. I also address AP’s rejection of the idea that fact-finders compute precise posterior probabilities of litigation positions. Although I agree with this point, I argue probabilism is better understood as a conceptual framework than a user manual. In the third section, I develop an additional similarity. Like RPE, probabilistic accounts can be used to understand cases in which the stories parties present to the jury do not exhaust the universe of logically possible explanations. In the final section, I offer some reasons to doubt AP’s claim that RPE solves the infamous conjunction problem.

In conclusion, I argue that comparative versions of the probabilistic account and RPE can work together, with RPE providing the positive foundation for a probabilistic conceptual account. Thus I see RPE less as a paradigm shift and more as an important friendly amendment to the best version of the probabilistic paradigm. One might say that RPE provides a basis for taking probabilism seriously, rather than literally.

Evidence, likelihood and posterior probability

Denote the evidence as X, and let A and B be two possible states of the world that are mutually exclusive but not necessarily exhaustive. Thus there might be some state C that both (i) is mutually exclusive with respect to $A$ and B and (ii) has positive probability of being true. To relate AP’s approach to probabilistic theories requires defining some probability objects, which I assume reflect the beliefs of some fact-finder named J (J might be a judge, a collective jury, or an individual juror).

Denote $P (A | X)$ as the probability that A is true when the evidence X has been observed—what is known as the posterior probability of A given X.² Also of interest is the ‘overall’ probability that A is true, $P (A)$ , which is the average of the posterior probability of A over all possible evidentiary states.

The Bayesian approach to evidence law turns on posterior probabilities of the state of the world given evidence presented at trial—e.g., objects such as $P (A | X)$ . A critical related probability is the likelihood of A given X, written $L (A | X)$ . The likelihood of A given X is not the same as the probability of A given X. Instead, it is the probability of X given A: $L (A | X) \equiv P (X | A)$ . This likelihood is a measure of the consistency of the data with the postulated state of the world: $L (A | X)$ tells us how frequently we’d observe X if we knew A really were the case.

As an example, consider a tort case that turns on whether D adequately shovelled the sidewalk. P alleges the contrary in a suit she filed after slipping and breaking her leg. Suppose X is the factual record composed of a photograph of the sidewalk, which P took shortly after her fall and which shows snow cover, together with D’s testimony that (i) she did shovel the walk, (ii) more snow fell thereafter, and (iii) the snow had stopped falling just a few minutes before P slipped. Let A include D’s position that she took adequate care. Then the likelihood of A given X is the probability of observing the photograph and testimony, including D’s, if D really did take adequate care. Because X is largely consistent with D’s case, the likelihood of A given X will be quite high in this case. But suppose now X also includes compelling proof that D is lying—say, a timestamped video of D somewhere else when she claimed to have shovelled the walk. Then it would be less probable to observe X when D has taken adequate care, so $L (A | X)$ would be smaller.

The posterior probability $P (A | X)$ and the likelihood $L (A | X)$ are closely related. Bayes’s Theorem states that

P (A | X) = L (A | X) \frac{P (A)}{P (X)},

where $P (X)$ is the overall probability of observing the evidence X. Equation (1) tells us that the posterior probability of A given X and the corresponding likelihood are proportional, with the proportionality factor being the ratio of overall probabilities. These overall probabilities are also known as ‘prior probabilities’ because they may be understood as capturing the fact-finder J’s beliefs about the probabilities of A and X prior to seeing any evidence.³

Prior probabilities are notoriously vexing. Typically we have at most limited basis for determining their values. Accordingly, Bayesian evidence law largely is concerned with subjective prior beliefs: each fact-finder is assumed to arrive at the court house with her own biases and beliefs, and these combine to form that person’s prior beliefs.

Bayes’s Theorem applies just as well to beliefs and probabilities related to state of the world B; just substitute ‘B’ for ‘A’ in equation (1). Because $P (X)$ appears in the denominator of both the resulting equation and equation (1) as written above, we can eliminate it by dividing the two equations. The result is⁴

\frac{P (A | X)}{P (B | X)} = \underset{\underset{Likelihood Ratio in favour of A}{︸}}{\frac{L (A | X)}{L (B | X)}} \times \underset{\underset{Prior Odds in favour of A}{︸}}{\frac{P (A)}{P (B)} .}

The ratio on the left-hand side of (2) is the posterior odds in favour of A (over B). To say that the posterior odds exceed 1 is to say that fact-finder J finds A more probable than B in light of evidence X. The posterior odds equal the product of the likelihood ratio in favour of A and the prior odds in favour of A. Thus, J’s posterior odds and likelihood ratio will be the same only when J thinks the prior odds are 1. In that case, J believes $P (A)$ and $P (B)$ are equal before seeing any evidence, so that J’s posterior odds in favour of A are entirely determined by the likelihood ratio in favour of A. Otherwise, posterior odds in favour of A and the corresponding likelihood ratio will differ.

Among probabilistic accounts of evidence law, it is useful to distinguish Bayesian and likelihoodist approaches. According to the Bayesian approach, a fact-finder applying the preponderance standard in distinguishing A and B, given evidence X, should find in favour of A whenever the posterior probability of A exceeds the posterior probability of B, i.e., whenever $P (A | X) > P (B | X)$ . This is equivalent to finding in favour of A whenever posterior odds exceed 1. The likelihoodist approach focuses instead on the likelihood ratio only. When the likelihood ratio in favour of A exceeds one, a likelihoodist will conclude that the evidence supports A more than B.

What distinguishes the two accounts is whether priors matter. According to likelihoodists, priors are irrelevant. Only the likelihood ratio matters, because all the information in observed evidence is encapsulated by the likelihood ratio. By contrast, Bayesian accounts take note of priors. A juror who walks into the court house thinking A is very unlikely will have low prior odds in favour of A. Consequently, even quite convincing evidence in favour of A, yielding a high likelihood ratio, will not always be enough to convince the juror to find in favour of A.

The case for Bayesianism is that jurors have biases—prior odds different from one—which positive models of fact finding must take into account. In favour of likelihoodism are the normative position that juror biases are disfavoured by the court system (consider voir dire), and the difficulty of measuring priors.⁵

The primary drawback to probabilism is simple: it’s tough to determine precise probabilities. So the approach’s appeal generally is only conceptual—much as a roughly drawn map does fine getting a driver from one place to another. This is what I meant when I wrote above that probabilism should be taken seriously but not literally. I lack the space to develop this argument in detail, but as I argue below, the overlap between probabilism and RPE is sufficient to view RPE as providing a positive basis for taking probabilism seriously.

The comparative strength of explanations and the distinction between conceptual usefulness and numerical particularism

According to AP, in the RPE approach ‘the central fact-finding task is not to attach probabilities to the individual elements’ (Allen and Pardo, 2019: 15). Rather, they explain, the central task according to RPE is ‘to determine whether potential explanations of the evidence and events satisfy the applicable standard of proof’ (Allen and Pardo, 2019: 15). Unlike an approach that requires attaching particular probability numbers to individual states of the world—A is true with probability 0.52—’the explanatory account is inherently comparative’, so that ‘whether an explanation satisfies the standard depends on the strength of the possible explanations supporting each side (and not only the party with the burden of proof)’ (Allen and Pardo, 2019: 15).

But the first section above indicates that neither probabilistic account is confined to ‘attach[ing] probabilities to’ particular states of the world. We can express likelihoodism in terms of the likelihood ratio in favour of A over B, and Bayesianism in terms of the posterior odds in favour of A over B.

I am not the first to make this point. Edward Cheng uses the approach outlined in the first section above to argue for representing the preponderance standard in terms of posterior odds.⁶ Addressing the contention that stories, rather than probability reasoning, better capture juries’ decision making, he argues that ratio approach in the first section above shows that ‘use of probabilistic tools and the story model are not as antithetical as they may first appear’ (Cheng, 2013). Whereas Cheng’s focus was largely (though not only) on positive matters, Cheng and Michael Pardo use the same ratio-based posterior odds approach in a normative context aimed at refuting various welfarist approaches to proof at trial (Cheng and Pardo, 2015). And Sean Sullivan uses a generalised likelihoodist approach that’s also fundamentally comparative.⁷

If the comparative approach doesn’t distinguish RPE and probabilism, what about AP’s insistence that it’s impractical to expect fact-finders to actually attach probabilistic numbers to each probability at issue in litigation?

Surely they’re right.

But probabilism’s conceptual usefulness doesn’t turn on this point. When the preponderance standard applies, a fact-finder need only determine whether, given the evidence, she thinks one or another state of the world is more probably the case. She might be able to do that even if she cannot compute the exact probability she places on each state. This is a broader point, one I don’t have space here to defend in detail, about the usefulness of conceptual models generally. What makes them models—rather than, say, portraits or user manuals—is their generality and their applicability as guides, and probabilism delivers on that dimension.

Cases where the contested probability space is less than the universe

When Plaintiff asserts story A, Defendant is within her rights to advocate simply ‘not-A’. In such cases, for Plaintiff to prove the posterior odds exceed one is equivalent to proving that the posterior probability of A exceeds one-half. Thus if Defendant simply denied Plaintiff’s account—if she placed the full universe of logical possibilities in issue—the conventional view of the preponderance standard would apply.

AP reject this view because they reject the idea that defendants will usually find it beneficial to take the ‘not-A‘ position. Instead, they argue defendants will usually do better by offering a more specific alternative story of their own. What defendants give up in probability units, they gain by framing a coherent explanation. To be extreme about it, it would be pretty silly to argue, in a case that turns on identity, that the plaintiff hasn’t proved that the Judge wasn’t the perp. No juror will believe that, and arguing it will make the defendant look silly, so it’s better to point to a particular alternative (the butler, say).

In such situations, only a restricted part of the universe is placed in issue at trial. AP’s argument here seems to hinge on the assumption that probabilistic accounts somehow cannot be based on such ‘universe restrictions,’ but this feature of trial strategy is not at all limited to RPE. To be formal about it, write the universe as $A \cup B \cup C$ , with Plaintiff arguing A, Defendant arguing B, and neither party addressing the residual part C. If all we care about is whether A or B has greater likelihood or posterior odds, then probabilistic accounts—whether likelihoodist or Bayesian—can work just fine using the ratio of probabilities involving only A and B. So the fact that $P (A \cup B) < 1$ is as immaterial to probabilisms as it is to RPE, unless probabilistic accounts are artificially prevented from reflecting practical trial strategy.

The conjunction problem

AP argue that RPE ‘avoids the conjunction paradox in a straightforward manner’ (Allen and Pardo, 2019: 18 (footnotes omitted)). To fix ideas, here’s a simple example of the conjunction problem at work:

To win, Plaintiff must establish elements E ₁ and E ₂ are both true.

Defendant argues the logical complement, i.e. Defendant argues that it is the case that not-E ₁, not-E ₂, or both hold.

The truth values of E ₁ and E ₂ are probabilistically independent.

For each element, the posterior probability that that element is true is 2/3.

Therefore the posterior probability that both elements are true is 4/9 (the square of 2/3).

What’s the problem? It’s that although there’s relatively high posterior probability that the evidence will show each element to be true when it is considered in isolation, still the posterior probability of both elements’ truth is less than one-half. So if the law requires only that each element be proved to a preponderance, then Plaintiff can win the case even though the posterior probability that her story—‘E ₁ and E ₂’—is true is less than the posterior probability of Defendant’s story—’not-(E ₁ and E ₂)’.

AP explain the resulting dilemma quite well:

The…law applies standards of proof to the elements of claims, crimes, and affirmative defenses…. [W]hen probabilistic thresholds (standards of proof) are applied to individual elements and not to their conjunction (the claim as a whole), then the probabilistic account no longer fits with the assumed goals of the standards of proof (i.e., regarding accuracy and the risk of error)…. According to one interpretation, the law is committing a devastating error and such an egregious mistake in the doctrine ought to be corrected. (Allen and Pardo, 2019: 13 (footnotes omitted))

AP offer a way out of what seems to be a normative problem, proposing that it really is a positive one:

Alternatively, however, the inconsistency may reveal that probabilistic thresholds are a poor way of explaining legal standards of proof. At a minimum, such thresholds fail to explain how current standards operate or how they align with the law’s goals. (Allen and Pardo, 2019: 13)

AP’s proposed fix can deliver on its promise only if no probabilistic account can possibly make sense. My argument here is similar to the one that AP themselves use to attack Kevin Clermont’s suggestion that the conjunction problem can be solved by replacing conventional probability with fuzzy set theory. AP point out that ‘changing one’s theory of probability does not change the world or one’s knowledge of the world’. (Allen and Pardo, 2019: 51) Clermont’s proposal fails, they argue, because ‘[t]he conjunction effect is [a] feature of the world (and a sticky one at that)—it will not change by swearing allegiance to fuzzy logic.’ (Allen and Pardo, 2019: 52)

Similarly, even if RPE is a better description of what fact-finders really do than hardcore probabilism, as long as the relevant probabilities are well defined objects, the conjunction problem exists. Unless I misunderstand their argument, AP’s claim that RPE resolves the conjunction problem therefore must be a claim that probabilistic accounts are necessarily false as a description of fact finding.

Let’s turn to AP’s argument that RPE does resolve the conjunction problem. Assume we have a civil claim with two elements, E ₁ and E ₂. AP write:

Rather than assessing E ₁ and E ₂ serially and attaching a probability to each, fact-finders evaluate whether the plaintiff’s explanation (which will include or entail E ₁ & E ₂) is better than the defendant’s explanation (which will fail to include E ₁ or E ₂, or both). Both legal elements come into play after an explanation has been selected (based on the explanatory threshold), in order to determine whether it includes the elements or not. Conceptualizing the relationship between the cases and elements in this manner ameliorates the ‘paradoxical’ consequences that arise under the probabilistic conception, and it provides an interpretation of the law that fits with the rationales underlying the standards of proof.⁸

Using my notation from above, call the plaintiff’s explanation A and the defendant’s explanation B. Following AP, elements E ₁ and E ₂ are true under A, and at least one is false under B. According to AP, the fact-finder must decide whether A or B is the ‘better’ explanation. But what does ‘better’ mean here? AP write that the ‘criteria [that] affect the strength or quality of an explanation…include considerations such as consistency, coherence, fit with background knowledge, simplicity, absence of gaps, and the number of unlikely assumptions that need to be made’ (Allen and Pardo, 2019: 16).

AP acknowledge a critique posed by Dale Nance—that these criteria ‘are natural criteria for the assessment of epistemic probability’ (Allen and Pardo, 2019: 20, citing Nance, 2016: 81). AP elaborate, explaining that ‘[a]ccording to our account, legal fact-finding does indeed aim at what are essentially probabilistic conclusions’ (Allen and Pardo, 2019: 20). Further, they write that ‘both relatively plausibility and the conventional probabilistic account are focused on the same end or goal’ (Allen and Pardo, 2019: 20), namely determining ‘what is more likely based on the evidence’ (Allen and Pardo, 2019: 20).

As we’ve seen, the conjunction problem exists whenever the truth-value of the elements E ₁ and E ₂ may be represented using the conventional laws of probability. So it is difficult to see how AP’s holistic approach can solve the conjunction problem.

But, AP explain, ‘we are relying on the fact that the inferences at trial are abductive’, and ‘[a]bduction…is the process by which legal fact-finders arrive at probabilistic…conclusions’ (Allen and Pardo, 2019: 20–21). Abduction is often referred to as ‘inference to the best explanation’.⁹ Granting that abductive inference is a better means of arriving at probabilistic conclusions than some competing alternative, if it is still a means of doing so, then I don’t see how it can solve the conjunction problem.¹⁰

And even if AP are right that jurors ‘do in fact proceed holistically—this is how information is processed’ (Allen and Pardo, 2019: 31), lots of consequential adjudication happens before a jury is ever empanelled (even when one has been demanded). Parties often move for summary judgment in ways that place discrete issues rather than whole cases at issue. When they do, judges trained in the law’s stated element-by-element approach must evaluate the record evidence with respect to individual issues. Perhaps they really do this ‘holistically’ even when they say they do so atomistically, but I wonder. Given the powerful role played by judges considering discrete issues at pre-trial summary judgment, more is needed to back up AP’s claim that RPE resolves the conjunction problem because of the holistic nature of trial fact finding.

Conclusion

There is plenty to like in the RPE framework that AP have developed. It provides an accessible and plausible positive foundation for fact finding at trial. Contra AP, though, RPE is conceptually consistent with probabilistic accounts of fact finding. Thus I see RPE not as a paradigm shift, but rather as an important and friendly amendment to well-conceived probabilism.

Footnotes

Acknowledgement

Professor of Law, University of Pennsylvania Law School. I thank Ronald Allen and Michael Pardo for inviting me to publish this comment on their very interesting paper, ‘Relative plausibility and its critics’.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Allen

Pardo

(2019) Relative plausibility and its critics. International Journal of Evidence and Proof 23(1-2): 5–59.

Bickel

(2012) The strength of statistical evidence for composite hypotheses: Inference to the best explanation. Statistica Sinica 22: 1147–1198.

Cheng

(2013) Reconceptualizing the burden of proof. Yale Law Journal 122(5): 1104–1371.

Cheng

Pardo

(2015) Accuracy, optimality and the preponderance standard. Law, Probability and Risk 14(3): 193–212.

Gelbach

(2018) Estimation Evidence (draft) 27 August 2018 (manuscript on file with author).

Gelbach

Kobayashi

(2018) Legal Sufficiency of Statistical Evidence. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3238793 (accessed 27 December 2018).

Kaplan

(1968) Decision theory and due process. Stanford Law Review 20: 1065.

Kaye

Sensabaugh

(2011) Reference guide on DNA identification evidence. In: Reference Manual on Scientific Evidence, 3rd ed. Washington, DC: National Academies Press.

Kaye

Freedman

(2011) Reference guide on statistics. In: Reference Manual on Scientific Evidence, 3rd ed. Washington, DC: National Academies Press.

10.

Nance

(2016) The Burdens of Proof: Discriminatory Power, Weight of Evidence, and the Tenacity of Belief. Cambridge: Cambridge University Press.

11.

Royall

(1997) Statistical Evidence: A Likelihood Paradigm. Boca Raton, FL: Chapman & Hall/CRC.

12.

Sullivan

(2019) A likelihood story: The theory of legal fact-finding. University of Colorado Law Review 90: 1.

13.

Zhang

(2013) A likelihood paradigm for clinical trials. Journal of Statistical Theory and Practice 7(2): 157–177.