We’ve had some astonishing examples of bad predictions by politicians and pundits lately. Let’s see how we can avoid making these mistakes in our own litigation forecasts by considering the habits of thought exercised by Tetlock’s superforecasters. This is Part 3 of a series by Robert Rothkopf, Managing Partner of Balance Legal Capital LLP, called Litigation Superforecasting, inspired by a recent book called “Superforecasting – The Art & Science of Prediction” by Philip Tetlock and Dan Gardner.
In Part 1, I suggested that litigation and litigation funding involves making predictions about future outcomes, and called for lawyers and their clients to embrace the use of percentage probabilities rather than vague verbiage (such as “significant chance”) in legal advice in order to reduce confusion and enhance the accuracy of forecasts.
In Part 2, I focused on the characteristics of Tetlock’s “superforecasters” and compared them to the stereotypical traits of lawyers.
In this Part 3, I will take a closer look at the methods and habits of thought used by Tetlock’s “superforecasters” that control for cognitive biases and improve accuracy, and suggest how lawyers might adopt them when formulating their own views on litigation.
A key finding of Tetlock’s data from large scale forecasting tournaments is that what distinguishes good forecasters from bad ones is simply a way of thinking. Superforecasters exhibit certain characteristics (see Part 2) and have habits of thought that tend to correct for the cognitive biases that lead our judgments astray. Tetlock seeks to summarise these in guidelines he calls the “Ten Commandments for Aspiring Superforecasters”.
Tetlock’s message is one of optimism: these habits of thought and the ability to make more accurate predictions are skills that can be cultivated.
In this article I discuss a few of Tetlock’s thought habits and apply them to litigation. These are steps that we have successfully incorporated into our due diligence and investment selection process at Balance Legal Capital.
1. Avoid vague verbiage and probabilistic language – use numbers – % probabilities
This was discussed in Part 1 of this series, and is the first habit to acquire in producing better litigation forecasts. Whenever you find yourself reaching for a probabilistic phrase in your litigation advice or when prognosticating about litigation outcomes, simply insert (in brackets, perhaps) a % number after the phrase.
So, instead of “There is a strong argument to support a finding of jurisdiction,” – try this: “There is a strong (65%) argument to support a finding of jurisdiction.” Numbers reduce confusion amongst writers and readers of advice, and the process of converting your thoughts into a number increases accuracy (through meta-cognition).
2. Unpack the question into ever smaller components – “Fermi-ize”
The next habit is to break up your case analysis into ever finer sub-components and then build it back up again. Tetlock calls this “Fermi-izing” – named after the engineer Enrico Fermi who developed this method of splitting a difficult unknowable question into discrete knowable parts.
For example, the over-arching question that a litigator advising his client is trying to answer is: “will my case win?”. This question cannot be answered without slicing it up.
Lawyers are generally trained to do this anyway, by breaking down causes of action into duty, breach, causation, loss and so on. Tetlock’s superforecasters showed that the further you break the question down, the more accurate the resulting estimate becomes. Are the necessary documents available? What could disclosure throw up? What are the chances that a witness will fail to perform well and what impact would a bad performance have on the case? How compelling is the evidence on causation of loss? And what evidence of loss is there? Each of these questions can be dissected further still.
Tetlock’s superforecasters were not only skilful at formulating such questions, but they were also good at turning questions around to consider other perspectives. In litigation, the other perspectives to consider are those of your opponent (and, specifically, their decision makers) and the judge. What does each defendant have to show to resist each claim or neutralize the loss? What will they do to discredit the witness or show that the claimant did not mitigate? How will the evidence be viewed by the judge or arbitrator? And who is the judge/arbitrator going to be? Considering each issue from your opponent’s perspective is a valuable check on the probabilities that you ascribe to your own case.
3. Harness the “anchoring” effect – start with the base rate, and then consider the particulars of the case
Tetlock calls this guideline “Strike the right balance between inside and outside views”. It covers three steps which combine to negate, or harness, the cognitive bias called the “anchoring effect”. We looked at this previously in Part 2. The anchoring effect is the phenomenon whereby individuals tend to rely too heavily on the first piece of information offered in formulating their views. For example, experiments have shown that simply displaying large numbers at random prior to posing a question such as “what is the population of Venezuela?”, significantly boosts the average guess. (It’s the reason why in most cases of negotiation on price, you do better to go first as you anchor your counterparty’s response.)
A. Consider the “base rate” / “outside view”
The outside view or “base rate” involves stepping away from the detail of the case, and asking yourself, and other experts in the field, what is the average or usual outcome for a case of this kind (i.e. for a particular cause of action / proceedings). By considering the base rate, you downplay the “uniqueness” of the case at hand and are more likely to “anchor” your ultimate estimate to an accurate starting point.
For example:
Whilst the same richness of data is currently not available for commercial litigation, there are businesses aiming to fill the gap, such as Amiqus.
B. Merge with the “inside view”
Having focused on the base rate, the second step is to consider the “inside view” – the detail of the case, and to consider whether the facts justify moving from the base rate, and by how much. What is so unique about this case to make you believe that it will fall into the 26% of ICSID cases where the claimant obtains an award in its favour or the 25.7% of cases that settle? And if a claimant is claiming losses of £100m, should their legal advisor temper their expectations by reference to the fact that applying the base rate would lead to a figure of £37m?
By focusing on the base rate numbers first, your brain places the particular facts of your case into a relevant perspective. The psychologist Daniel Kahneman and Tetlock’s superforecasters have shown that this method produces more accurate estimates than if you had jumped straight into the detail immediately.
4. Seek multiple perspectives – “dragon fly forecasting”
Tetlock’s superforecasters were good at integrating multiple points of view and sources of information in reaching their estimates. Tetlock calls this “dragon fly forecasting”. It is connected to the theory of “the wisdom of the crowd” which says that it is impossible for one person to know everything necessary to reach an accurate view – relevant information is dispersed among different sources and if you consult a large number of people, the valid information compounds, and the errors cancel each other out. This is why a crowd asked to guess the number of sweets in a jar gives a remarkably accurate estimate if you take the average of their answers.
Forecasting the outcome of litigation is clearly more complex than guessing the number of sweets in a jar. However, dragon fly forecasting is still a useful technique. At Balance, if a case looks likely to meet our criteria, it warrants multiple members of the team reviewing the key materials independently. We document our assumptions and our thought processes that drive our estimates – i.e. what base rates are available, which of the inside view characteristics we thought were compelling, and the Fermi-ized questions we posed. Where appropriate, we also seek views on discrete points from our network of solicitors, barristers and quantum experts – and incorporate them (weighted accordingly) into our estimates.
5. Update forecasts as new information comes to light
Tetlock’s forecasting tournament data showed that forecasters who frequently updated their forecasts in light of new information achieved higher forecasting scores. Interestingly, the superforecasters who made small adjustments, up or down, of an average of 3.5% on each revision, did better. However, this of course depends on the gravity of the new information. A high value settlement offer from the defendant or an adverse document uncovered in disclosure may warrant larger revisions of a prospects of success estimate than a more trivial development.
This may sound trite, but it emphasizes the value of following relevant legal and financial developments outside of the case, as well as the evidential and procedural developments inside it, and of updating your forecasts accordingly. The act of updating a merits opinion regularly will compel the individual providing the opinion to stand back and consider the issues afresh in light of up-to-date information and to adjust the case strategy if necessary.
6. Keep score
Finally, lawyers should be keeping track of prospects advice they give and the manner and terms upon which their clients’ disputes are resolved. By doing this, lawyers can compare their predictions against actual outcomes, and examine when they are wrong, and when they are right and why this was so. This is the only way to improve.
It can be time-consuming and challenging to apply to litigation the habits of thought that Tetlock’s superforecasters have honed. But like all skills, these get easier with practice.
For litigation funders and litigators alike, it is difficult to resist the temptation of jumping straight into the details of a new case when a client rings, and risk wrongly anchoring your views and that of your team.
Like us, lawyers should keep score, tracking whether their forecasts on a given case proved to be right or wrong. Law firms might even find that it becomes a useful marketing tool if they can boast about the accuracy of their litigation forecasts on budget, duration and outcome.
Finally, Tetlock’s findings also highlight the benefit of consulting with a litigation funding team comprising experienced litigation and arbitration lawyers like Balance Legal Capital, who can bring a fresh perspective and apply superforecasting techniques to your case.
Take the Litigation Superforecasting Survey: Put a number on it
Balance is proud to share that we are supporting the Justice and Equity Centre (formerly the Public Interest Advocacy Centre) in its pursuit of strategic social justice litigation.
Balance is proud to support Velitor Law in their pursuit of landlords and insurance brokers for secret commissions.
Balance is proud to have been ranked again in the Chambers and Partners 2024 Litigation Support Guide in both the UK and Australia.
Robert Rothkopf, has been invited to speak at The Commercial Litigators Forum panel event discussing The Post Office Case on 12 June 2024.
Jeremy Humm will be taking part in the upcoming Thought Leaders 4 FIRE Webinar “Litigation Funding Post-PACCAR – What has happened since the decision and what happens next?”
Balance is a member of the Association of Litigation Funders of England & Wales. Our membership of the ALF requires us to satisfy on an ongoing basis stringent financial and capital adequacy tests and to comply with the ALF Code of Conduct.
Balance is a founding member of the Association of Litigation Funders of Australia, and a member of ILFA, the International Legal Finance Association.
Balance Legal Capital LLP is authorised and regulated by the UK’s Financial Conduct Authority.
.png)
.png)
