An evidence problem for family justice: social work use of transcription AI, harmful hallucinations and the deletion of original audio

Social worker use of transcription AI may have led to some efficiencies for social workers, but it is impacting the integrity of written documents, with significant risks downstream for family justice.

Recent Ada Lovelace Institute research has identified harmful hallucinations from these technologies in real-world social work cases, including one AI-generated summary which stated a person had expressed suicidal ideation when they had not.

If a family court were to receive such a hallucination in social work evidence, they would be hamstrung to determine it to be as such, since a standard is fast emerging amongst social work data retention policies that these tools are configured to delete the original audio in 30 days, thereby losing an authoritative piece of evidence by which the court could determine what is authentic and what is hallucinated.

JUSTICE, the human rights and law reform charity, is advocating for a systems-approach to arriving at agreed standards for AI tools which will impact the fairness of court proceedings, rather than these standards being decided unilaterally by the commercial vendors. This includes data and audio preservation and auditability standards. This work has started in criminal justice by police and the CPS: their joint operating procedure highlights the importance of retaining original audio when AI is used for transcription in a criminal context.

JUSTICE recommends the right standards for family proceedings – which take into account the need to preserve audio for the purpose of authenticating AI-generated transcripts, but also consider data limitation and practical storage capacity constraints – should be determined through system-wide discussions, including the perspectives of the judiciary, social workers, people affected by care processes, and regulators who establish standards for social work.

Background

There has been rapid expansion of the use of transcription AI across public services in the UK in the last year or so, including by social workers: in early 2025, one AI transcription tool – Magic Notes from private company, Beam – was already in active use by 85 local authorities for social care.

Transcription AI is distinct from more traditional methods of automated transcription due to its incorporation of generative AI in the form of large language models (LLMs). LLMs help fill in gaps and infer content when audio quality is poor or contains long silences. However, by doing so, modern AI transcription tools have integrated a hallucination risk as a design choice to improve performance: there is an increased likelihood of plausible text output because of the LLM, but also an increased likelihood of fabrications.

The risk of AI hallucination

The rapid adoption of these technologies in social work has not been matched with greatly increased research and evaluation. To help plug the gap, the Ada Lovelace Institute conducted qualitative research, published in February this year, based on conversations with frontline social work practitioners using AI to transcribe and summarise their interactions. The majority of social workers they spoke to were using either Microsoft Copilot or Beam’s Magic Notes.

The research found the tools can deliver efficiencies, however it also identified a risk of harmful hallucinations and misrepresentations of people’s experiences being entered into statutory care records. For example, one social worker noted, after an interaction with a client, that an AI summary incorrectly “indicated that there was suicidal ideation”, but “at no point did the client actually, you know, talk about suicidal ideation or planning, or anything.”

The research also highlighted inconsistency in safeguards, challenges to existing accountability models, and an absence of impact evaluation and monitoring beyond efficiency, for example there is very little evidence on how AI transcription tools impact the experiences and outcomes of people who draw on care.

JUSTICE’s concerns and recommendations

Along with recommendations for better research, JUSTICE particularly also agrees with the need for clarity and guidance on using these tools in statutory processes and formal proceedings.

In producing such guidance, a systems-approach is needed, which considers the downstream impact of AI-transcribed records coming into evidence in statutory proceedings.

It is clear that no AI transcription tool will be 100% accurate, and there are good arguments that this should not be a bar to use, since of course human notetakers are not failsafe. JUSTICE therefore suggests the question is: if harmful mistakes do make it into care records or care assessments which are then submitted into court proceedings, what safeguards are needed to ensure the court can do its job and ensure fair and safe proceedings?

The court process should enable fair challenge to the local authority evidence, including the accuracy of any transcription in evidence. Applying the best evidence principle, and some common sense, JUSTICE observes the courts will be best placed to resolve any factual dispute between parties over the accuracy of any AI transcribed documentation by hearing the original audio recordings. These will simply elucidate if the transcribed notes are accurate or not. Eg in the example in the research report above, if suicidal ideation was mentioned or if it was hallucinated by the AI tool.  

The court’s ability to access best evidence in exercising its fact-finding role however will be hampered if that original audio does not exist. Surprisingly, a standard is emerging amongst social work use of these tools: the tools are often configured to automatically delete the original audio in 30 days.

This is potentially a fatal flaw when it comes to safeguards since it undermines the court’s function – to allow for fair challenge and the ability for inaccurate evidence to be rectified where it matters most, when what is said may have legal consequences. For example, if suicidal ideation is relied upon as evidence of threshold being met in a care case, the absence of an audio recording undermines the court's ability to do its job in the event of a dispute, i.e. the parent saying "I didn't say that”.

There is also a risk of the opposite happening, namely that it becomes beneficial for litigants to make spurious allegations of inaccuracy of care records, in the knowledge that no authoritative audio recording exists (known as the liar’s dividend).  

Notably, not retaining audio is at odds with what has been agreed as best practice in the criminal justice context. The CPS and National Police Chiefs’ Council have agreed a Joint Operating Procedure (JOP), which highlights the importance of retaining source information, and inputs (i.e. audio recordings).

Next steps

JUSTICE has voiced the above concerns in a workshop hosted by the Ada Lovelace Institute on 15 June 2026. Some participants agreed, whilst others raised various problems with retaining audio recordings for AI transcriptions, including data protection concerns and concerns about capacity and security of data storage solutions. JUSTICE does not consider any of these problems nullify JUSTICE’s concerns, nor its identified solution, however they must of course be taken into account.  

It is unrealistic to expect any change in these tools to be instigated by the vendors. They are made for the market, and any change in design or configuration will be a response to the demands of their customers. Indeed, Beam contacted us following the first version of this blog to explain that customers are able to configure the tool to longer periods of retention if they choose: 30 days is a customer choice, not a fixed design feature. Yet our review of publicly available policies suggests that 30 days is fast becoming the “norm” for social work data retention configuration, without any wider thinking taking place about whether that is the best for all those involved. Again, this is not for Beam, or Microsoft, or any other developer, to pre-empt. Those customers – social workers – need an agreed set of standards for the use of AI transcription in social work which reflects system-wide perspectives of what safeguards are needed, including safeguards for the courts to do their job.

JUSTICE suggests this work is urgently required. System-wide discussions should take place to identify these standards, and JUSTICE of course recommends that audio retention is considered as a critical piece of the puzzle. Such discussions should incorporate the perspectives of the judiciary, people affected by care processes, and regulators who establish standards for social work.

‍

This blog was edited on 16.09.26 to reflect the following:

1. The majority of social workers spoken to in the Ada Lovelace Institute’s research used either Microsoft Copilot or Beam’s Magic Notes. Paragraph 8 previously just referred to practitioners “using AI” to transcribe. Elsewhere in the blog, Magic Notes was mentioned by name, but Copilot was not. Copilot has therefore been mentioned explicitly to correct any misunderstanding that social workers spoken to only used Magic Notes.
2. The Ada Lovelace Institute research did not specify which AI tool made the harmful hallucination of suicidal ideation. The previous version wrongly attributed this to Magic Notes when no such attribution is made in the research – paragraph 9 has been edited accordingly.
3. A standard is emerging amongst social work use of these tools: they are often configured to automatically delete the original audio in 30 days. However the previous version of the blog suggested they were “designed” this way. Beam have informed us that customers are able to configure the tool to longer periods of retention if they choose: 30 days is a customer choice, not a fixed design feature. Paragraphs 15 and 20 have been edited to correct any misunderstanding.