In the late 1990s, when I was working at Stirling University, I did some research on the accuracy of umpires' leg-before-wicket (LBW) decisions in cricket. This work was published as
Craven, B.J. (1998) A psychophysical study of leg-before-wicket judgments in cricket. British Journal of Psychology, 89, 555-578.
The work was the subject of an article by Raj Persaud in the Daily Telegraph.
I also wrote the following article for a more general audience. It was published in How's That?, the journal of the Association of Cricket Umpires and Scorers, Issue 263, June 1999.
No rest for the wicket
How to make 500 LBW decisions in one hour
The qualities needed for a good umpire are many: sharp eyesight, concentration, endurance, sound knowledge...and a thick skin. The man in the middle is required to make difficult judgments about complex events that are over in fractions of a second, with resentful players and carping commentators ever ready to point out and discuss any supposed faults in their decisions. The LBW decision is one of the more complex umpiring decisions, not just because of its multi-faceted nature, but because it requires the umpire to make a judgment not just about what events did happen, but about what events would have happened had not the batsman's leg interrupted the flight of the ball. Not surprisingly, LBW decisions are a rich source of controversy.I work in the psychology department of Stirling University, and ply an obscure trade known as visual psychophysics in an effort to find out more about how our sense of vision works. Obviously, the LBW decision is above all a visual task, and I noticed that the techniques that I was using to investigate other aspects of human vision were perfectly suited to finding out how good umpires really are at LBW decisions. In fact, they would also work for line calls in tennis, or any other situation where an official has to make a yes-no decision based on the position of a ball at a certain instant.
Using a computer, I created a 3-D umpire's-eye view of a cricket pitch, complete with stumps and markings. To this scene, the computer added an animated view of a cricket ball going down the pitch as if it had been bowled. The physics of the flight and bounce of a cricket ball has been quite thoroughly studied, so I could be confident that my simulated ball was behaving as a real cricket ball would. One thing that proved beyond my wit to depict accurately was a batsman, so instead I had a simple white rectangle, the size of a batsman's pad, appear in a suitable place as the ball made its journey from one end of the pitch to the other. The ball was seen to travel down the pitch, bounce, strike the pad, and rebound away, bouncing once or twice before it rolled to a halt. The task of the "umpire" was to say whether the ball would have passed to the left of the wicket, would have hit the wicket, or would have passed to the right of the wicket. During the experiment, the subject would make thousands of decisions like this, with the ball following a slightly different trajectory each time. After each decision, the subject was shown whether the ball really would have hit the stumps. Potentially, they were able to use this feedback to improve their performance. As neither of my subjects were umpires, I felt that it was important to give them as much chance to benefit from practice as possible, so that I could find out what are the limits on human performance at the LBW task.
I could have done a similar experiment using real balls, real wickets, and a bowling machine. I chose to use the simulation because it gave me absolute control over the nature of the flight of the ball, absolute certainty about whether the ball would have hit the stumps had it not hit the pad, and the ability to get my subjects to make thousands of LBW decisions in a short time. Of course we have to be a little bit cautious about extending results obtained in this way to the real situation.
How can we describe how good an umpire is?
In effect, the umpire is making a measurement of where the ball would have been in relation to the stumps as it passed the plane of the wicket, if it hadn't been intercepted by the pad. To give an informative description of how good a measurement is, we need to concern ourselves with two things: consistency and bias.
Consistency
Imagine an umpire presented with a long sequence of LBW decisions, in which the ball, in reality, would have missed the off stump by 1cm if it hadn't struck the pad. The human brain is never quite the same from one moment to the next, and so the umpire's judgments will vary. Sometimes the umpire will perceive the ball as missing the wicket by more than 1cm, (from now on I will miss out the phrase "had it not been intercepted by the pad") sometimes they will perceive the ball as missing the wicket by less than 1cm. If the variation in their judgments is large enough, they will sometimes perceive the ball as hitting the wicket. If we can say that the ball-to-ball variation in an umpire's judgments is x cm, we have said something useful about how good that umpire is. In everyday terms, we might say that the umpire can judge the future trajectory (flight path) of the ball to the nearest x cm.
Bias
This is not the same as favouritism! Imagine a fantastically consistent umpire, whose judgments were consistent to the nearest tenth of a millimetre. We would have complete faith in this umpire's judgments, right? Wrong. The umpire might, with great consistency, always perceive the ball to be ten centimetres to the right of where it is really travelling! When making LBW decisions about the off stump, this umpire would have a lot of disgruntled batsmen to deal with. This systematic misperception of the ball's trajectory is what I call bias. An analogy would be a very good archer who has set the sight on their bow wrongly - their arrows might cluster very tightly together, but nowhere near the centre of the target.
An umpire could have any combination of consistency and bias. The could be wildly inconsistent, but unbiased. They could be highly consistent, and biased as well, and so on.
How well do umpires cope with swinging deliveries?
Bias is particularly interesting in the context of cricket because of the things that bowlers do to make things difficult for the batsman. Imagine a right-handed batsman facing a delivery that swings away. The batsman must decide what shot to play very early in the flight of the ball, and so must forecast the future trajectory of the ball. If the batsman's forecast doesn't allow for the swing on the ball, they will play inside the line. That is, their idea of where the ball is is biased: they play as if it were a few centimetres to the left (as they look at it) of where it really is. The bowler's aim when they swing the ball is to induce a bias in the batsman.
Now a very similar thing could happen to the umpire. It could be that, if the bowler causes the ball to swing, the umpire would be unable to allow for the swing on the ball, and would, for example, have a tendency to mistakenly give batsmen out for balls that were swinging away just outside the off stump. In one of my experiments, I presented my subjects with a mixture of deliveries: some swung to the left, some swung to the right, and some were straight; the subject didn't know which kind of delivery was coming next. I measured the bias for each kind of delivery.
The results were impressive. One of my two subjects was scarcely affected by the swing of the ball, showing the same bias of less than 5mm throughout. The other subject did appear to be affected by swing on the ball, but even then the bias was small: only greater than 1cm for balls that swung more than about 12cm between bowler and wicket. Overall, he appeared to be able to allow for roughly 95% of the swing on the ball. These biases really are tiny. For example the diameter of the ball is about 7cm.
How consistent are umpires?
I measured consistency at the same time as measuring bias. The two subjects behaved rather differently, but overall, they judged the position of the ball as it passed the wicket to the nearest 2-3cm. Now you could have a rather pointless argument about whether this performance is impressive or regrettable, but the really interesting result is that one of the subjects started the experiment judging the trajectory to the nearest 4cm, but several thousand decisions later, they were judging it to the nearest 2cm. In other words, they improved with practice. The other subject, who had a lot of practice previously as the experiment was being developed, judged to the nearest 2cm throughout.
What's the upshot for umpires?
After all that detail, it would be worth summarising the important results. Firstly, my subjects were rather good at allowing for swing on the ball. Secondly, at least one of the subjects was able to use feedback to increase the consistency of their decisions. Where do these results lead us?
The third umpire, etc
One unfortunate side-effect of my work on LBW decisions is that it draws attention to the errors that umpires make. As a result, some people assume that the inevitable conclusion is that umpires on the field should be replaced by umpires watching video recordings, or by some sort of elaborate technology involving cameras and computers, and no doubt a laser or two thrown in for good measure.
I don't see that this follows at all. It is true that umpires make errors, but it is also true that the ball sometimes bounces erratically off cracks in the pitch, and that the arbitrary timing of a shower of rain can decide a Test Match. Nobody wants to play on pitches like billiard tables, and I don't suppose that many people want to play cricket under a roof, either. So why, when we are happy to accept these random influences on the game, shouldn't we be equally happy to accept the slight vagaries of umpires' decisions?
We also ought to remember that people play and watch cricket not to discover which is the best team, but to enjoy themselves. Who is to say that replacing umpires with technology would make the game any more enjoyable for anyone apart from the odd prima donna batsman or bowler?
The LBW law
One thought that we should always bear in mind is that the LBW law is only there to work around a fault in the design of the game. In a game where the batsman is meant to use one straight stick to defend three other straight sticks, the necessary presence in the immediate vicinity of two other straight sticks (in the form of the batsmans' legs) is nothing but a nuisance. The spirit of the LBW law is to discourage the use of the legs as a means of defence; for various reasons the letter of the law allows the batsman to pad up with impunity in some cases.
If we think that experiments like mine tell us something useful about the performance of umpires at LBW decisions, we could apply the same techniques to the other aspects of the LBW decision (line of pitching, line of interception by the pad, etc). Suppose that we found that one of these sub-decisions is disproportionately difficult compared to the others. Could we perhaps design an LBW law that doesn't ask umpires to make the sub-decision that is responsible for most of their errors, while remaining true to its intended purpose?
Training
The finding that practice with feedback can improve consistency offers an opportunity for umpires. It is true that umpires do get practice at present, by actually being umpires. But this practice is limited (an umpire would have to be rather busy to make 1000 LBW decisions in a year) and umpires only get feedback to the extent that batsmen, bowlers, or television pundits voice their disagreement. Now imagine a "flight simulator" for umpires that runs on a home computer, which would allow them to make thousands of LBW decisions in a few hours, with full accurate feedback about the results of each decision. Something similar could be done for run-outs, too, though catches are more tricky. The knowledge that umpires have a means of objectively "calibrating" themselves might help to restore some of the respect that, not through their own fault, they appear to be slowly, and sadly, losing.