A study published in “Cell Reports Physical Science” unveiled a machine-learning classifier that can detect when the AI chatbot ChatGPT has authored academic chemistry papers. The study, released on November 6, demonstrates that this specialized tool outperforms existing artificial intelligence (AI) detectors and could become an essential resource for academic publishers to identify submissions created by AI text generators.
The Drive for Specialized Detection
Heather Desaire, a chemist at the University of Kansas in Lawrence and co-author of the study, stresses the value of specificity in developing AI detectors. “Most of the field of text analysis wants a general detector that will work on anything,” Desaire explains. “We were going after accuracy by making a tool that focuses on a particular type of paper.”
This precision-targeted approach has yielded a system that offers enhanced performance by concentrating on the unique characteristics of academic writing within the chemistry discipline.
Training AI to Recognize AI
The research team initially described their ChatGPT detector in June, focusing on Perspective articles from the journal “Science”. The detector uses machine learning to analyze 20 writing style features, such as sentence length variation and the frequency of specific words and punctuation. The researchers created a highly accurate detector by training the tool with a combination of human-written text and introductions generated by ChatGPT-3.5 in the style of American Chemical Society (ACS) journal articles.
When tested against human and AI-generated text, the detector identified introductions written by ChatGPT-3.5 with 100% accuracy when provided with paper titles and 98% accuracy with abstracts. Impressively, the tool maintained its efficacy with text from the newer ChatGPT-4.
Superior Performance Over Competitors
In comparison, the ZeroGPT AI detector only achieved a 35–65% accuracy rate, and an OpenAI-developed text classifier had a success rate of merely 10–55%. The new tool also showed adaptability, performing well with introductions from journals outside its training set and against prompts designed to confound AI detectors. However, its specialized nature means it didn’t recognize human-written articles from university newspapers as such.
A Step Beyond Stylometrics
The approach taken by the authors is hailed as “something fascinating” by Debora Weber-Wulff, a computer scientist at HTW Berlin University of Applied Sciences. Traditional tools have focused on predictive text patterns, but applying stylometrics to AI like ChatGPT, is a novel and effective strategy.
Addressing the Root Causes
Weber-Wulff cautions that while the detection tool is impressive, it doesn’t tackle broader issues in academia, such as the pressure on researchers to rapidly produce papers or undervalue the writing process in scientific work. She warns against viewing AI detection tools as a panacea for what are fundamentally social problems within the scientific community.
As AI continues to evolve, so must the tools we use to ensure the integrity of academic publishing. Developing specialized detectors like the one from Desaire’s team represents a significant step forward in identifying AI-generated content. However, it also highlights the need for a broader conversation about the role of AI in academia and the pressures that drive its use.
