In a case that may mark the beginning of a troubling trend, a defence submission in a Florence courtroom cited false information generated by ChatGPT, OpenAI’s conversational AI tool. The incident, which took place earlier this year, saw an Italian lawyer include fabricated case law in a legal defence—content sourced directly from the chatbot.
The presiding judge quickly identified the error and dismissed the reference, raising concerns about the reliability of AI-generated content in legal contexts. Although the damage in this particular case was minimal, it highlights the growing risks of AI hallucinations—false or invented information presented with confidence by language models.
This is not the first time ChatGPT has found itself at the centre of legal controversy. In 2023, a U.S. attorney made headlines for citing non-existent case law produced by ChatGPT, leading to court sanctions. However, the Florence case may be the first high-profile example within the European judicial system, setting off alarm bells across the continent.
Legal experts warn that as AI becomes more embedded in research and documentation, due diligence is critical. “ChatGPT can assist with drafting and ideation, but it cannot replace verified legal sources,” noted a professor of law at the University of Milan. “The burden of verification still lies with the professional using the tool.”
At the heart of the issue is the illusion of authority AI tools can create. The chatbot’s ability to generate coherent, persuasive text makes it easy to mistake fiction for fact, particularly when under pressure or lacking cross-referencing skills. With generative AI adoption growing in law firms and corporate legal departments, the risk of AI-induced misinformation entering courtrooms is increasing.
The Florence judge’s sharp eye may have saved this particular case from proceeding on faulty grounds, but experts fear many similar cases may go unnoticed—leading to potential miscarriages of justice. Calls are now growing for clearer regulation and AI literacy training across the legal profession.
The courtroom, once a bastion of rigorous fact-checking and precedent-based reasoning, now faces a new challenge: keeping pace with AI.
Discussion about this post