Sounding Right, Being Wrong: The Growing Threat of AI-Generated Court Pleadings
Sounding Right, Being Wrong: The Growing Threat of AI-Generated Court Pleadings
Courts in Florida and across the country are grappling with a rapidly growing crisis: the submission of court filings produced with the assistance of generative artificial intelligence that cite nonexistent cases or cite actual cases for inaccurate legal propositions. As the Florida Supreme Court recently noted: “Generative artificial intelligence tools—including large language models and other systems that produce text or citations in response to prompts—are being used as drafting or research aids for court filings. Though these tools can be helpful, they also can generate content that appears plausible but is in fact inaccurate, including fabricated or ‘hallucinated’ authorities.” In re: Amendments to Florida Rule of General Practice and Judicial Administration 2.515, No. SC2026-0673 (Fla. May 28, 2026).
This problem is occurring with increased frequency, and courts are finding themselves on the front lines of a battle to preserve the integrity of the judicial system. The Fourth District Court of Appeal succinctly described the issue in addressing a frivolous argument in a recent brief: “This Court has seen this AI-hallucinated argument often. Seeing it at all is too often.” Avery v. Beauzil, Case Nos. 4D2026-0406 and 4D2026-0921 (Fla. 4th DCA May 27, 2026).
What Is Generative AI and Why Is It Dangerous?
Generative artificial intelligence programs compile data to generate statistically probable outputs when prompted. These programs often rely on language models trained on billions of parameters, making it virtually impossible to determine how a program came to a specific result. Critically, generative AI can hallucinate or create inaccurate answers that sound convincing. According to one recent study, generative AI programs for legal research hallucinate legal authorities between seventeen and thirty-three percent of the time. See Russell v. Mells, 426 So.3d 913, 918 n.3 (Fla. 2d DCA 2025).
As articulated in a concurring opinion from Judge Lott of Florida’s Fourth District Court of Appeal, AI chatbots “do not ‘think.’ They make predictions about what words ought to come next in response to a prompt that the user provides.” Gouveia v. Meridian Fin. Invs., LLC, — So.3d —-, 2026 WL 816497, at *3 (Fla. 4th DCA Mar. 25, 2026) (Lott, J., concurring). Generative AI “is very good at sounding right, but less adept at being right, especially where critical thought is required in creation of the content.” Id. This creates a dangerous dynamic in the legal context: the output looks like a properly formatted legal brief, complete with case citations and quotations, but the underlying authorities may be entirely fabricated.
The Pro Se Litigant Problem
Judge Lott further discussed why pro se litigants represent a particularly acute dimension of this crisis. Unlike attorneys, pro se litigants “are usually not repeat players in the court system,” have “little experience or knowledge on how to litigate cases,” and do not appreciate “how or why a response might not be right.” Id. Because AI-generated content can look and sound authoritative, pro se litigants “put it in their brief to see what happens.” Id. And because “the cost to generate the content is so low, they can put in a lot of it,” leaving “the opposing party and the court … in the position of breaking down why something that sounds right is not right, which tends to consume more resources than parsing through a traditional pro se appeal.” Id.
Many recent cases involving pro selitigants illustrate this problem vividly, including citations to non-existent cases and citations for legal propositions not actually present in the cited cases. See Hessert v. Hessert, — So.3d —-, 2026 WL 785016, at *1 (Fla. 6th DCA Mar. 20, 2026); Gutierrez v. Gutierrez, 399 So.3d 1185, 1187–88 (Fla. 3d DCA 2024); Gouveia v. Meridian Fin. Invs., LLC, — So.3d —-, 2026 WL 816497 (Fla. 4th DCA March 25, 2026); Gleason v. Marcus, — So.3d —-, 2026 WL 1311959, at *1 (Fla. 2d DCA May 13, 2026).
Attorneys Are Not Immune
The crisis is not limited to pro se litigants. Licensed attorneys have also submitted AI-generated fabrications to courts. In Russell v. Mells, an attorney submitted an appellate brief containing misquotations of two cases and a citation to a case that did not exist and appeared to have been “hallucinated” by a generative AI program. See Russell, 426 So.3d at 918-19. When the court issued an order to show cause, the attorney acknowledged that her citations “were researched via computer generated searches” and that she “failed to fully vet these searches.” Id. at 919. When the attorney proposed that because “the substance of the analysis in her brief wasn’t necessarily wrong, her misstatements are not an issue we should be overly concerned about,” the court found this explanation deeply troubling and called this position “simply unacceptable.” Id.
The Second District Court of Appeal emphasized that “[e]thical requirements are not excused simply because a computer program generated a faulty or misleading legal analysis,” nor is it “an excuse that the attorney did not intend to mislead the court.” Id. at 920. “To state the obvious, it is a fundamental duty of attorneys to read the legal authorities they cite in appellate briefs or any other court filings to determine that the authorities stand for the propositions for which they are cited.” Id. (internal citations omitted).
What Opposing Counsel Should Look For
Courts have provided clear guidance on what attorneys should watch for in filings from opposing parties—whether represented by counsel or proceeding pro se. The Sixth District Court of Appeal directed attorneys that “if and when any such erroneous filings are made in a given case before our Court, filers on the opposite side of the case should point out such errors to our Court either in their responsive filings or via motion.” Hessert, 2026 WL 785016, at *2.
Practitioners should be particularly alert to common hallmarks of AI-generated filings. In addition to looking for citations to cases that do not exist and quotations attributed to cases that do not contain the quoted language, be wary of briefs that sound authoritative but contain an unusually high number of unfamiliar authorities, abnormal formatting, and charts or tables that can appear out of place.
Sanctions and Consequences
Courts have imposed a range of sanctions for the misuse of generative AI in filings. The Third District Court of Appeal dismissed an appellant’s consolidated appeals as a sanction and barred him from future pro se filings related to the underlying probate actions unless reviewed and signed by a member of the Florida Bar. Gutierrez, 399 So.3d at 1188. The Fourth District Court of Appeal issued similar sanctions against a pro se litigant, directing the clerk to reject any future filings unless signed by a member of the bar and requiring a notice of appearance from a member of the bar in the litigant’s pending cases to avoid dismissal. Avery, Case Nos. 4D2026-0406 and 4D2026-0921 (Fla. 4th DCA May 27, 2026). The Second District Court of Appeal referred an attorney to the Florida Bar for potential disciplinary action, noting that “[w]hen a lawyer cites imaginary legal authorities to our court as if they were law, we are compelled to refer that lawyer to the Bar because of the professional rules of conduct.” Russell, 426 So.3d at 920. The Sixth District Court of Appeal issued an order to show cause directing a petitioner to demonstrate why she should not be sanctioned for her defective petition. Hessert, 2026 WL 785016, at *3.
Court Recommendations and Emerging Standards
Several principles and expectations have emerged from the growing body of case law addressing generative AI misuse. First, courts have set forth expectations that litigants and attorneys must cite-check all cases cited in their filings to ensure they actually exist and actually represent the legal propositions asserted, regardless of whether such filings are prepared with or without the assistance of AI. Second, courts have made clear that pro se litigants and attorneys “cannot satisfy their aforementioned obligation to cite-check all cases cited in their filings by relying upon AI to cite-check the cases cited in their filings.” Hessert, 2026 WL 785016, at *2.
Third, courts have emphasized that attorneys who delegate work and use technological tools such as generative artificial intelligence in representing their clients remain responsible for the work product that is generated.Russell, 426 So.3d at 921. As the Fourth District Court of Appeal stated: “Artificial intelligence may assist in research and drafting, but it cannot replace the lawyer’s or litigant’s duty to ensure that every authority cited is real, accurate, and applicable. The responsibility for what is filed rests with the human author, not the machine.” Francois v Vive Financial LLC, — So.3d —-, 2026 WL 758289 at *1 (Fla. 4th DCA March 18, 2026)
At least eight of the twenty judicial circuits in Florida have issued circuit-wide administrative orders requiring attorneys and self-represented litigants to include an AI disclosure or certification when filing a document prepared with assistance of AI. The Fifth, Seventh, Eighth, Eleventh, Fifteenth, Seventeenth, Eighteenth, and Nineteenth Judicial Circuits issued orders with generally similar wording that the filer used generative AI in the preparation of the document, and that all assertions, legal authority, and citations have been independently verified for accuracy. The administrative order from the Seventeenth Judicial Circuit further requires that the filer identify the specific tool used in creating the document.
Perhaps most notably, on May 28, 2026, the Florida Supreme Court issued an Order amending Florida Rule of General Practice and Judicial Administration 2.515. The amendment requires the signer of a document filed in Florida courts to represent that “the legal authorities identified exist and are accurately cited.” The amendment becomes effective June 15, 2026 and also provides for the imposition of sanctions that may include reprimand, contempt, striking of the document, dismissal of proceedings, costs, and attorneys’ fees.
Conclusion
The message from Florida courts is clear: generative AI is a powerful tool, but it is not a substitute for professional judgment, diligence, and candor. Practitioners would be wise to heed the warnings and orders, both by ensuring the integrity of their own filings and by remaining vigilant in scrutinizing the filings of opposing parties.
