Australian courts have been dealing with the consequences of lawyers turning to AI to draft legal submissions, only to find out the AI had entirely made up case citations. One widely reported case involved an immigration lawyer who filed court documents containing 17 non-existent authorities, all generated by ChatGPT. The Federal Circuit and Family Court referred the matter to the NSW Office of the Legal Services Commissioner, with the judge warning about the risks of relying on “unverified” AI-generated content without proper checks or independent verification (The Guardian). This case and others like it have sparked broader discussion within the legal profession about the role of generative AI in legal practice, and the importance of lawyers upholding their professional obligations, particularly the duty of competence and the duty to the court.
A simple test
To test these issues, we asked ChatGPT to generate a short 150-word summary explaining the Australian copyright law principle of “substantial part” in copyright infringement cases, including case references. Here’s what it produced:
“In Australian copyright law, infringement can occur even where only a portion of the work has been reproduced, provided that the portion amounts to a “substantial part.” The test focuses on quality over quantity—even a small excerpt may infringe if it captures essential elements of the original work. The question is whether the part taken embodies the author’s skill, labour, or originality.
This principle was applied in Autodesk Inc v Dyason (No 2) (1993) 176 CLR 300, where the High Court held that copying even a small section of computer code constituted infringement because it replicated a critical component of the program. Similarly, in EMI Songs Australia Pty Ltd v Larrikin Music Publishing Pty Ltd (2011) 191 FCR 444, the court held that a short musical phrase from Kookaburra reproduced in Men at Work’s Down Under was a substantial part, despite its brevity.”
In this instance, the AI performed relatively well. Both cases are real and correctly cited:
- Autodesk Inc v Dyason (No 2) is a complex High Court decision involving copyright in computer software where the High Court ultimately reaffirmed their ruling in Autodesk Inc v Dyason (No 1) and determined that copying a functional but original look-up table amounted to reproducing a substantial part of the computer program.
- The key issue in EMI Songs v Larrikin Music was whether Down Under, the iconic Australian anthem by Men at Work, infringed the copyright in the song Kookaburra. The court determined that the Down Under recordings reproduced a substantial part of the song, which constituted copyright infringement.
The summary itself was fairly accurate. It explained the core principle clearly (within the limits of 150 words): that Australian courts focus more on the quality than the quantity of what has been copied when assessing whether a “substantial part” has been taken. This was at the heart of the controversy in Autodesk, where the High Court wrestled with the tension between originality and whether something’s functional importance could also make it substantial, albeit minimal. ChatGPT correctly pointed to two of the most well-known cases in this area.
However, this was a relatively safe topic, relying on well-known cases. AI tools tend to produce far more errors and even fabricate cases when asked about more obscure areas of law or lesser-known precedents.
A more complex test
We decided to try ChatGPT on another question: “Write 100 words with case citations on the recoverability of pure economic loss in NSW law”. The response was:
“Under New South Wales (NSW) law, the recoverability of pure economic loss is generally limited. Pure economic loss refers to financial loss not resulting from physical damage to a person or property. In Demerara Distillers Ltd v. Fair Trading [2018] NSWCA 266, the Court held that pure economic loss is typically not recoverable in negligence unless there is a special relationship between the parties, as seen in Sullivan v. Moody [2001] HCA 59. Courts apply a restrictive approach to prevent indeterminate liability, adhering to the principle that recovery for economic loss should be confined to situations with proximity or reliance.”
Looks plausible? As far as we can tell, the first cited case doesn’t exist, and the case citation belongs to an unrelated case – though there is an interesting-looking Jamaica Fair Trading complaint involving Demerara Distillers and the geographical indication for “Jamaican Rum”.
The second case does exist at its citation, but the run-on sentence suggests that it includes a determination on pure economic loss, which is not the case – it’s about the existence of a duty of care. In the relevant recent High Court decision Mallonland Pty Ltd v Advanta Seeds Pty Ltd [2024] HCA 25, the first case is not mentioned, and Sullivan is mentioned in the context of evaluating the existence and scope of the duty, rather than in the context of pure economic loss.
What does ChatGPT itself say about its legal knowledge?
When asking ChatGPT what its general legal knowledge was on Australian case law, this is what came up:
“My legal knowledge is current up to mid 2024, including Australian law. However, it can access the most recent legal cases, news, or legislative updates through web search in real time.”
That is one of the risks that courts and regulators have been warning about. Anything newer would still need to be researched on legal databases to ensure accuracy.
What about client confidentiality?
AI tools are certainly useful for summarising documents and even creating plain-language versions of advices. But the warning is on the label: “Don’t share sensitive info. Chats may be reviewed and used to train our models.” So, lawyers and consultants must take care not to upload material without first redacting anything that is confidential to the client.
AI can be useful as a starting point for legal research, or early drafting, but human review remains non-negotiable. While these tools can offer a starting point, they should not replace thorough legal analysis or proper source-checking, no matter how convenient the technology might seem.
Author: Ashna Govil, graduate lawyer.
