Page 68 - Reforming Benefits Decision-Making

effective scrutiny of UC, which will help contribute to public support and
187
confidence in the system.
2.88 A number of consultees we spoke to raised questions or concerns about what
the Intelligent Automation Garage is doing. There is very little information
about it in the public domain. The DWP told us that to date it has built and
deployed 50 automations. These are applied to mundane processes and
replace rules-based processing steps. However, this could cover a whole range
188
of processes, many of which may be important to decision making.

2.89 One of the issues in the Johnson and Pantellerisco cases was a lack of
transparency about what the actual constraints of the UC system and other
computer systems it interfaces with are, and the ability to adapt the UC
system to deal with unforeseen issues caused by the automation.

2.90 There is legitimate concern amongst claimants and civil society organisations
regarding the use of automation and artificial intelligence in fraud detection
and prevention. Privacy International sent a series of FOIA requests to the
DWP asking for further information about the use of artificial intelligence for
fraud prevention referenced in the DWP’s annual report. The DWP refused to
provide information on the criteria or indicators used by the system to flag
someone as likely to be committing fraud, or the code of the algorithm being
used. They relied on FOIA exemption that release would prejudice the
189
prevention and detection of fraud and crime.
SyRi

The SyRI scandal in the Netherlands demonstrates the dangers of automated
fraud detection for individual rights and the need for transparency. A fraud-
detection system (named SyRI) was suspended in 2020 after having falsely
accused more than 20,000 parents of child benefit fraud. Parents who were

187 R. Pope, Universal Credit: Digital Welfare (see n. 56 above) p. 100-101.
188 The only processes that we are aware that have been automated are pensions claims. See Convedo,
‘How Intelligent Automation is improving public services’.
189 Privacy International, ‘Shedding light on the DWP Part 2 – A Long Day’s Journey Towards
Transparency’ (February 2021).

59