An artificial intelligence (AI) system used by the Department for Work and Pensions (DWP) to detect welfare fraud is showing “statistically significant” disparities related to people’s age, disability, marital status and nationality, according to the department’s own internal assessment.
Released under freedom of information (FoI) rules to the Public Law Project, the 11-page “fairness analysis” found that a machine learning (ML) system used by the DWP to vet thousands of universal credit benefit payments is selecting people from some groups more than others when recommending whom to investigate for possible fraud.
Carried out in February 2024, the assessment showed there is a “statistically significant referral… and outcome disparity for all the protected characteristics analysed”, which included people’s age, disability, marital status and nationality.
It said a subsequent review of the disparities present found “the identified disparities do not translate to any immediate concerns of discrimination or unfair treatment of individual or protected groups”, adding that there are safeguards in place to minimise any potentially detrimental impact on legitimate benefit claimants.
“This includes no automated decision-making,” it said, noting that “it is always a human [who] makes the decision, considering all the information available”.
It added that while protected characteristics such as race, sex, sexual orientation, religious beliefs and so on were not analysed as part of the fairness analysis, the DWP has “no immediate concerns of unfair treatment” because the safeguards apply to all customers. It plans to “iterate and improve” the analysis method, and further assessments will be completed quarterly.
“This will include a recommendation and decision on whether it remains reasonable and proportionate to continue operating the model in live service,” it said.
Caroline Selman, a senior research fellow at the Public Law Project, told the Guardian: “It is clear that in a vast majority of cases, the DWP did not assess whether their automated processes risked unfairly targeting marginalised groups. DWP must put an end to this ‘hurt first, fix later’ approach, and stop rolling out tools when it is not able to properly understand the risk of harm they represent.”
Because of redaction, it is not currently clear from the analysis released which age groups are more likely to be wrongly targeted for fraud check by the AI system, or the differences between how nationalities are treated by the algorithm.
It is also unclear whether disabled people are more or less likely to be wrongly singled out for investigation by the algorithm than non-disabled people. While officials said this was to stop people from gaming the system, the analysis itself noted any referral disparity related to age (particularly for those 25 and over) or disability specifically is anticipated because people with these protected characteristics are already linked to a higher rate of universal credit payments.
Responding to the Guardian report, a DWP spokesperson said: “Our AI tool does not replace human judgement, and a caseworker will always look at all available information to make a decision. We are taking bold and decisive action to tackle benefit fraud – our fraud and error bill will enable more efficient and effective investigations to identify criminals exploiting the benefits system faster.”
While the assessment outlined the measures the DWP has put in place to mitigate any potential bias – including that the model will always refer claimant requests it defines as high risk to a DWP employee, who will then decide to whether or not to approve it – the assessment did not mention anything about the role or prevalence of “automation bias”, whereby users are more likely to trust and accept information generated by computer systems.
Computer Weekly contacted the DWP about whether it had assessed dynamics around automation bias within the operation of the AI system, and if so how this is affecting referral and outcome disparities, but received no response by time of publication.
The role of AI and automation in welfare systems has come under increased scrutiny over recent months. In November 2024, for example, an analysis by Amnesty International found that Denmark’s automated welfare system creates a barrier to accessing social benefits for certain marginalised groups, including people with disabilities, low-income individuals and migrants.
The same month, an investigation by Lighthouse Reports and Svenska Dagbladet found that Sweden’s algorithmically powered welfare system is disproportionately targeting marginalised groups for benefit fraud investigations.
