DIGITAL LIFE
New study reveals that AI cannot fully write like a human
A world's first study shows that AI-generated writing continues to display distinct stylistic patterns that set it apart from human prose. Led by researchers at University College Cork (UCC), the research explores whether systems such as ChatGPT can genuinely write in a way that is indistinguishable from people.
The study is the first in the world to use literary stylometry, computational methods traditionally used to identify authorship, to compare the writing styles of humans and large language models like ChatGPT across creative texts.
Researchers found that AI can generate polished, fluent prose, but its writing continues to follow a narrow and uniform pattern. Human authors display far greater stylistic range, shaped by personal voice, creative intent, and individual experience.
Published in Humanities and Social Sciences Communications, the research contributes to a growing conversation about the future of creative work at a time when generative AI is reshaping education, publishing and the arts.
Distinct stylistic differences between human and AI writing...Led by Dr. James O'Sullivan, UCC School of English and Digital Humanities, the study marks the first time literary stylometry methods have been used in a systematic way to assess stylistic overlap between humans and AI.
The study has delivered one of the most detailed assessments to date of how large language models differ from human authors in their approach to storytelling. Dr. O'Sullivan analyzed hundreds of short stories written by people alongside pieces generated by AI systems. By examining subtle linguistic markers, including the frequency of common words, the research reveals clear and consistent stylistic differences.
The analysis shows a clear stylistic divide between human writing and text generated by large language models. AI systems like GPT-3.5, GPT-4, and Llama 70B produce tightly grouped clusters, each reflecting the uniform patterns typical of a given model. In contrast, human texts show far greater variation and individuality.
'Even when ChatGPT tries to sound human, its writing still carries a detectable fingerprint'...GPT-4 writes with even more consistency than GPT-3.5, but both remain distinct from human work. While GPT-3.5 occasionally comes close to human style, these moments are rare. Across both the clustering and MDS visualizations, the pattern is the same: AI models produce compact, predictable styles, while human writing remains more varied and idiosyncratic, traits that reflect individuality and creative intention.
Dr. O'Sullivan said, "While AI writing is often polished and coherent, it tends to show more uniformity in word choice and rhythm. In contrast, human writing remains more varied and idiosyncratic, reflecting individual habits, preferences and creative choices."
"Even when ChatGPT tries to sound human, its writing still carries a detectable fingerprint, which suggests that computers and people don't yet write in quite the same style," Dr. O'Sullivan explains.
What stylometry can and can't tell us about AI writing...The researchers caution against using stylometry as an AI detection tool in education but highlight its value in understanding how human expression differs from algorithmic generation, offering new insights into what makes writing recognizably human.
Dr. O'Sullivan said, "Stylometry can reveal broad patterns across large bodies of text, but it has no place in judging authorship in education. Students' writing shifts from task to task and is shaped by context, support, and lived experience, which makes stylometric detection both unreliable and ethically questionable in academic integrity cases."
The study shows that the most advanced models still fall short of writing creative prose that blends seamlessly with human work. The findings point to the need for broader datasets, new prompts, and testing with emerging models, as well as closer attention to the ethical and creative questions raised by the growing use of generative AI.
Dr. O'Sullivan said, "It is one thing for a large language model (LLM) to reliably produce an email or summary report; the capacity to automate the production of literature raises profound ethical and philosophical concerns about authenticity, originality, and the very nature of authorship."
Professor John F. Cryan, UCC Vice President for Research and Innovation, said, "Congratulations to Dr. James O'Sullivan on this timely study on comparing the writing styles of humans and AI driven language platforms. The staggering expansion and influence of AI in our daily lives is a defining feature of the 21st century. This research exemplifies the creative, collaborative, and interdisciplinary nature of research at UCC through our UCC Futures framework, in this case the areas of UCC Futures—AI and Data Analytics and the Future Humanities Institute."
Provided by University College Cork
