LITIGATION ADVICE INVOLVES PREDICTING THE FUTURE

At Balance Legal Capital we predict litigation outcomes and invest capital accordingly. Put simply, we try to answer the following question: what is the probability that this case will result in a positive outcome for the client and a return for our investors?

This question is not peculiar to litigation funders. It is broadly the same question that clients ask their litigation counsel. As litigators and funders know, predicting litigation outcomes is not simple. A multitude of complex factors need to be evaluated and integrated in order to forecast the result. Clients want lawyers to provide confident predictions, while lawyers generally hide behind carefully caveated advice and hedged forecasts.

A recent book by Philip Tetlock (Professor of Psychology and Management at the Wharton School, University of Pennsylvania) and Dan Gardner provides upbeat and valuable lessons for all those involved in predicting litigation outcomes. The book is called “Superforecasting – The Art & Science of Prediction”. It outlines the cognitive biases that we are all susceptible to, drawing on the research of legendary psychologists Daniel Kahneman and Amos Tversky, and combines those observations with data from large scale forecasting tournaments in which participants were asked to predict the likelihood of a range of political and economic issues. The results reveal that many “experts” think they are great forecasters but in fact are lousy. What divides good forecasters from bad forecasters is actually a way of thinking rather than innate gifts. The data shows that anyone with the right mindset can train themselves to pick up the habits of thought that improve forecasting accuracy.

In this three-part series, I will extract some important lessons from “Superforecasting” that litigators and consumers of legal advice can apply to improve the accuracy of their litigation forecasts. After all, better forecasts mean better advice, better decisions and more efficient allocation of resources for clients, lawyers, and the wider judicial system.

In this part 1, I set the scene by highlighting the need for precise language and percentage figures in litigation opinions rather than the vague probabilistic phrases that lawyers tend to prefer when predicting the outcomes of a dispute. In part 2, I will summarize the characteristics of good forecasters identified by Tetlock and compare them to the stereotypical traits of litigators. Finally, in part 3, I will survey Tetlock’s methods for “superforecasting” and suggest how these can be applied to litigation.

MISUNDERSTANDING THROUGH PROBABILISTIC LANGUAGE

Clients typically want a binary answer to their key question – “Will my case win? Yes or no?”.

Historically, lawyers have been reluctant to forecast litigation outcomes. Experienced litigators know that nothing in litigation is certain – the case may involve untested points of law, difficult factual issues, unhelpful documents can surface in disclosure and a witness may have a bad day. Depending on the stage of the litigation, very limited information may be available. All of these factors need to be considered and evaluated. The exercise is difficult and distilling the answer into an absolute “yes” or “no” is not realistic. Lawyers are wary of giving hostages to fortune and therefore tend to use probabilistic phrases that provide wiggle room in case the outcome does not follow the prediction: phrases such as “more likely than not”, “on balance”, “strong prospects”, “a fair chance”, “serious possibility”.

But what do these phrases mean and how useful are they to stakeholders in a dispute? Has a code emerged amongst lawyers where “more likely than not” means greater than 50%? How much greater than 50%? Does the phrase “strong prospects” mean about 60%? A “near certainty” – 95%?

Multiple studies have shown that people attach very different meanings to “probabilistic language”. Tetlock uses these memorable examples:

1. In the lead up to the disastrous 1961 Bay of Pigs invasion of Cuba, a military adviser told President John F Kennedy that the mission had a “fair chance” of success. While the adviser had intended the phrase to mean that the invasion had 3 to 1 odds against success, JFK interpreted “fair chance” to mean a much more positive assessment.
2. Sherman Kent, a Yale University history professor and pioneer at what became the CIA, issued a national intelligence estimate that said an attack by the Kremlin on Yugoslavia in 1951 was a “serious possibility”. Kent intended the statement to mean there were 65:35 odds in favour of an attack taking place. However, State department officials interpreted it to mean much lower odds. Kent then found that people in his own team producing the estimate had also interpreted the phrase differently – some thought it meant 4:1, others thought it meant the opposite.

Therefore, using percentage probabilities in merits opinions, such as “65%” or “70% chance of success”, instead of probabilistic language, serves to clarify what the forecaster means.

IMPROVING ACCURACY BY USING PERCENTAGE PROBABILITIES

Some solicitors and barristers resist providing percentage probabilities of success in their legal opinions and regard the exercise as an artificial one.

However, in addition to reducing the chance for miscommunication discussed above, Tetlock draws on studies that show that using numbers rather than vague language actually increases the accuracy of a forecast. This is because the forecaster is forced to consider carefully how he or she is thinking – a process called “meta cognition”. In so doing, the brain is more likely to identify and control for cognitive biases, to switch from over-relying on intuitions and instinct (or ego), and to engage more rigorously with the evidence.

USERS OF LEGAL ADVICE

Users of legal advice need to carefully consider the nature of the advice they are seeking and the way they should interpret legal opinions. Clients may want confident, concrete advice to support their decisions. However, studies have shown that people confuse confidence with competence. Middling judgments are taken to mean the forecaster has been too lazy to expend the effort required to gather the information that would justify greater confidence.

But the process of making a good forecast requires an acknowledgment that there is an irreducible uncertainty in predicting anything. Nothing is 100% or 0%. Probabilistic thinking is required.

In the context of complex commercial litigation, clients should be pleased to see a well-thought-through percentage chance of success that only very tentatively steps outside of the 35 – 65% range when there are cogent reasons to do so. Furthermore, users of legal advice that includes a percentage chance of success need to understand that it is just an estimate based on information available at the time of the forecast. A 70% chance of success means that there is a 30% chance of failure. If the client’s case was run 100 times, he would win 70 times and lose 30 times. A case with a forecasted 70% chance of success that loses does not necessarily mean that the forecast was wrong.

CONCLUSION

Lawyers find it difficult to give percentage probabilities of success in their merits opinions. However, the forecasting tournament data discussed by Tetlock in “Superforecasting” shows that the sooner we start getting comfortable with using percentage figures, and comparing our forecasts with the end results, the better we will become at forecasting. Better forecasts mean better case decisions, and better allocation of resources for lawyers, clients, and the justice system.

Read Part 2 in this series – where I examine the characteristics of Tetlock’s “superforecasters” and compare them to the stereotypical traits of litigators.

Take the Litigation Superforecasting Survey: Put a number on it.

Image ‘Airborne’ by Paul Kehrer, Flickr