Ministers need to “immediately” update the law to acknowledge that computers are fallible or risk a repeat of the Horizon scandal, legal experts say.
In English and Welsh law, computers are assumed to be “reliable” unless proven otherwise. But critics of this approach say this reverses the burden of proof normally applied in criminal cases.
Stephen Mason, a barrister and expert on electronic evidence, said: “It says, for the person who’s saying ‘there’s something wrong with this computer’, that they have to prove it. Even if it’s the person accusing them who has the information.”
Mason, along with eight other legal and computer experts, was invited by the government to suggest an update to the law in 2020, following a high court ruling against the Post Office, but the recommendations they submitted were never applied.
He and colleagues had been expressing alarm about the presumption as far back as 2009. “My view is that the Post Office would never have got anywhere near as far as it did if this presumption wasn’t in place,” Mason said.
The legal presumption that computers are reliable stems from an older common law principle that “mechanical instruments” should be presumed to be in working order unless proven otherwise. That assumption means that if, for instance, a police officer quotes the time on their watch, a defendant cannot force the prosecution to call a horologist to explain from first principles how watches work.
For a period, computers lost that protection in England and Wales. A 1984 act of parliament ruled that computer evidence was only admissible if it could be shown that the computer was used and operating properly. But that act was repealed in 1999, just months before the first trials of the Horizon system began.
As a result, when post office operators were accused of having stolen money, the hallucinatory evidence of the Horizon system was deemed sufficient proof. Without any evidence to the contrary, the defendants could not force the system to be tested in court and their loss was all but guaranteed.
The influence of English common law internationally means that the presumption of reliability is widespread. Mason cites cases from New Zealand, Singapore and the US that upheld the standard and just one notable case where the opposite happened.
In 2007, a Toyota Camry on an Oklahoma highway accelerated suddenly and stopped responding to controls, eventually crashing and killing a woman and seriously injuring a second. It was one of a rash of claims at the time that Toyota’s vehicles were experiencing uncontrolled acceleration, but thanks to panicked messages from the car’s passengers, one of the only ones with hard evidence suggesting an electronic malfunction.
“As little as a single bit flip can cause the driver to lose control of the engine speed in real cars due to software malfunction that is not reliably detected by any fail-safe,” one expert witness in the case said. Toyota eventually settled to avoid punitive damages, after a jury awarded $3m in compensation to two women.
The rise of AI systems made it even more pressing to reassess the law, said Noah Waisberg, the co-founder and CEO of the legal AI platform Zuva. “With a traditional rules-based system, it’s generally fair to assume that a computer will do as instructed. Of course, bugs happen, meaning it would be risky to assume any computer program is error-free.
after newsletter promotion
“Machine-learning-based systems don’t work that way. They are probabilistic … you shouldn’t count on them to behave consistently – only to work in line with their projected accuracy.”
Waisberg added: “It will be hard to say that they are reliable enough to support a criminal conviction.”
James Christie, a software consultant who co-authored the recommendations for an update to the law, said the changes could come in two stages. “The first would require providers of evidence to show the court that they have developed and managed their systems responsibly, and to disclose their record of known bugs.
“If they can’t … the onus would then be on the provider of evidence to show the court why none of these failings or problems affect the quality of evidence, and why it should still be considered reliable.”
The Ministry of Justice has been contacted for comment.
