Page 68 - JUSTICE and AJC Report - Reforming Benefits Decision-Making

replace rules-based processing steps. However, this could cover a whole range
of processes, many of which may be important to decision making.
188
2.89 One of the issues in the Johnson and Pantellerisco cases was a lack of
transparency about what the actual constraints of the UC system and other
computer systems it interfaces with are, and the ability to adapt the UC
system to deal with unforeseen issues caused by the automation.

2.90 There is legitimate concern amongst claimants and civil society organisations
regarding the use of automation and artificial intelligence in fraud detection
and prevention. Privacy International sent a series of FOIA requests to the
DWP asking for further information about the use of artificial intelligence for
fraud prevention referenced in the DWP’s annual report. The DWP refused to
provide information on the criteria or indicators used by the system to flag
someone as likely to be committing fraud, or the code of the algorithm being
used. They relied on FOIA exemption that release would prejudice the
prevention and detection of fraud and crime.
189
SyRI

The SyRI scandal in the Netherlands demonstrates the dangers of automated
fraud detection for individual rights and the need for transparency. A fraud-
detection system (named SyRI) was suspended in 2020 after having falsely
accused more than 20,000 parents of child benefit fraud. Parents who were
identified as having committed fraud had their benefits cut and were told to
repay everything they had ever received. Those who did not repay sufficiently
quickly were required to also pay fines. 190 Many of those affected were from
an immigrant background and the errors forced hundreds into severe financial
difficulty. 191

188 The only processes that we are aware that have been automated are pensions claims. See Convedo,
‘How Intelligent Automation is improving public services’.
189 Privacy International, ‘Shedding light on the DWP Part 2 – A Long Day’s Journey Towards
Transparency’ (February 2021).
190 The Economist, ‘A benefits scandal sinks the Dutch Government’ (see n. 185 above).

191 BBC, ‘Dutch Rutte government resigns over child welfare fraud scandal’ (January 2021).

59