Detecting ChatGPT: How AI Essays Differ from Student Work

As the distinction between human-created and AI-generated content becomes increasingly indistinct, identifying the source of a piece of work is more challenging than ever. To address this issue, LingualNeeds has conducted an in-depth analysis of AI’s influence on essay writing and learning in today’s educational landscape.

Over half of UK undergraduates have acknowledged utilizing AI in their essays, and U.S. colleges and high schools have reported a staggering 76% increase in assignments featuring AI-generated content within just one year. This trend highlights students’ growing awareness of how these innovative tools can enhance their learning experiences. Despite the mixed reviews surrounding AI, many professors are recognizing the value of generative AI tools as valuable aids in research and academic work.

To explore this trend, LingualNeeds conducted a comprehensive analysis of over 12,000 essays, comparing human-written and computer-generated content. Our study highlights the most common words, phrases, and errors found in both types of writing. The essays analyzed include student submissions sourced from the IvyPanda writing database alongside those crafted by ChatGPT.

We reached out to professors at the University of Edinburgh to explore the effects of AI on academic studies and to investigate the feasibility of identifying AI-generated content in student assignments.

So, what patterns can we see when analyzing student-written essays against AI-generated text? Here’s what we found.

Key takeaways

• Student-written essays showed a high error rate, with 78% containing at least one mistake, while only 13% of AI-generated essays had errors.
• The most common punctuation errors in student essays included missing commas, hyphen errors, and missing spaces.
• Students frequently used the words “people,” “also,” “one,” and “however,” while AI essays predominantly featured terms like “social,” “cultural,” “individuals,” and “health.”
• ChatGPT utilizes 35% fewer unique words than humans, employing just 7,308 unique terms compared to 11,248 found in student essays.
• Humans typically prefer simple nouns and verbs such as “one” or “people,” whereas ChatGPT tends to use more complex and specialized vocabulary like “cultural” or “economic.”

Online lessons

Learn a language with personalised 1-on-1 classes with our expert tutors from all over the world

English lessons online
French lessons online
Spanish lessons online

What are the most common errors in student and AI-generated text?

No matter how much you study with top-notch professors or take private English lessons, human errors in essay writing are inevitable. In contrast, AI-generated text tends to have significantly fewer spelling and grammar mistakes, making it a reliable alternative.

Our analysis revealed a significant disparity in error rates between human and AI writing. In a review of 12,000 student essays, we identified over 45,000 errors, averaging three mistakes per essay. In stark contrast, the 115 essays generated by ChatGPT contained just 15 errors, translating to only one error for every ten AI-written pieces. This highlights the remarkable accuracy of AI compared to human writing.

To identify which school or department has the highest error rate, we analyzed the number of mistakes by subject. This approach helps us pinpoint where students are most prone to making errors in their assignments.

We encourage science department students to frequently proofread their assignments, as research shows they tend to make the most mistakes. On average, each science essay contains about five errors, likely due to the complex terminology involved. Similarly, essays in economics, tourism, and religion also exhibit a high number of mistakes. By prioritizing proofreading, students can improve their writing and reduce these errors.

In contrast, essayists focusing on the environment, sociology, and media demonstrated the fewest errors in their work. Despite this, their essays still contained an average of three mistakes each. This highlights the importance of thorough proofreading, no matter what subject you’re tackling.

While identifying those who make the most mistakes is straightforward, we also explored the common types of errors, focusing on punctuation and phrasing missteps.

Our analysis revealed that the most frequent punctuation mistake in essays was the omission of commas, with at least one comma missing in each piece. In fact, over 40% of the essays we examined showed this error. Following closely were issues with hyphens and missing spaces, highlighting a significant area for improvement in punctuation skills.

Generic phrasing mistakes were the most frequently flagged by proofreading tools, followed by repetitive words.

Did AI write this? Comparing words in AI and student writing

How does ChatGPT’s writing style differ from that of humans? By examining the unique word usage in student essays versus those generated by ChatGPT, we uncovered intriguing patterns that highlight these differences.

Our research reveals a striking difference in language diversity between student-written essays and those generated by ChatGPT. In fact, ChatGPT employs 35% fewer unique words. While students showcase an impressive vocabulary with over 11,000 distinct words, technology-generated essays only incorporate about 7,000. This highlights the richer linguistic variety found in human writing.

The term “social” emerges as a common thread in both human essays and AI-generated texts. This important adjective is frequently highlighted in discussions about human interactions and social behavior, indicating a shared focus in both human and machine-generated research.

The distinctions between these two data sets are significantly greater than their similarities. In human writing, the most commonly used terms are broad, featuring a mix of generic nouns, adjectives, and pronouns that balance content and function. In contrast, the ChatGPT data set emphasizes ideas and concepts such as ‘culture’ and ‘economics,’ often addressing social issues. Additionally, the AI predominantly uses the transitional word ‘moreover,’ while human writing typically includes ‘also’ and ‘however’ as well.

To delve deeper, we analyzed the ratio of words used in human language versus those in ChatGPT’s responses. Our goal was to identify any potential biases favoring specific categories of words that might be more prevalent in one source than the other.

Interestingly, the word “would” appears significantly more in student essays than in those generated by ChatGPT. In fact, for every 217 instances of “would” used by humans, the AI only incorporates it once. This modal verb plays a crucial role in articulating hypothetical scenarios and expressing speculation—elements commonly found in essays but often challenging for AI to convey effectively.

When comparing the two lists, distinct differences emerge. Students predominantly favor concrete nouns and verbs such as “man” and “country,” while ChatGPT tends to gravitate toward abstract ideas like “identities” and “boundaries.” This results in contrasting tones: students typically employ straightforward vocabulary, whereas AI often opts for more ambiguous and figurative language.

Learn online

Learn a language online with our free learning materials.

Learn Spanish online
Learn English online
Learn French online

The most common phrases in student essays

Student writers often rely on common phrases such as “in order to” and “based on” to enhance their sentences. These prepositional markers help clarify their arguments and guide readers through their analyses. This tendency is particularly evident in the qualitative aspects of the data, highlighting that individuals tend to elaborate more when writing long-form content.

In addition to prepositions, connecting conjunctions such as “as well as” and “is one of” frequently appear in essays to enhance arguments and elaborate on ideas. These phrases serve to create more complex sentences and seamlessly introduce new points, which is a hallmark of effective essay writing.

What trends can we observe in longer sentences and phrases? We find a recurring use of explanatory and connecting expressions. This preference for providing context in essays enhances clarity and supports arguments, making it clear why students frequently favor these structures.

This is evident in phrases such as “it is important to know that,” which emerged as the most frequently used extended expression. Similarly, phrases like “it is possible to say that” and “this is due to the fact that” are popular methods for elaborating on ideas within sentences.

The phrases that vary slightly are “in different parts of the world,” which serves as a geographical reference, and “the purpose of this paper is to,” a common phrase found in the introductions of genuine essays.

While it’s reassuring that humans excel at directing and summarizing for readers, incorporating a wider range of phrasing can enhance originality and prevent your writing from blending in with others.

How do you spot AI-generated text? Asking the experts

Curious about professors’ perspectives on AI? We reached out to Dr. Vaishak Bell, a Reader at the School of Informatics, and Professor Judy Robertson, Chair in Digital Learning at the University of Edinburgh. They shared their insights on the growing influence of AI and its effects on student research today.

Is there a danger in students using AI for essay writing?

Dr. Bell highlights a significant challenge we face today: some systems are capable of generating inaccurate information. If individuals haven’t thoroughly examined the subject matter, it can be incredibly challenging to identify these errors.

The rise of tools like ChatGPT presents a unique challenge for both students and examiners. Many students may believe they can find shortcuts to grasp complex concepts, leading to ongoing struggles in their understanding. However, the insights generated by AI can often be superficial or even misleading. Without a solid foundation of knowledge, students may find it difficult to discern these inaccuracies.

Prof. Robertson is focused on eliminating the temptation to cheat with AI tools. He questions why students would refrain from using AI if it is allowed and acknowledged in their work. To address this, the marking scheme includes penalties for excessive reliance on AI and for incorporating misinformation. However, the most significant points are awarded for original ideas that showcase human creativity, encouraging students to present innovative work that stands out.

Would you be able to detect AI writing or not?

Dr. Bell explains that the appropriateness of writing style largely depends on the context. In scholarly writing, for instance, a passive voice is commonly used, and adjectives are typically avoided to keep the focus on factual content without embellishment. He notes that he has observed ChatGPT’s rephrasing and found the differences to be minimal.

I believe we can’t truly surpass it solely through syntactic structure. Instead, we need to delve into a semantic analysis of the topic. This approach makes the task quite challenging, and I don’t think it’s an easy feat to accomplish.

Prof. Robertson highlights an intriguing challenge in education: even seasoned teachers struggle to differentiate between human-written and AI-generated essays when relying solely on a database of work from unknown authors. However, in his small class, where he knows each student’s unique voice and their individual progress, he feels more equipped to identify the nuances in their writing. This familiarity allows him to discern subtle differences that might reveal whether an essay is genuinely theirs or crafted by an AI.

I believe that genuine human connection is our best defense against AI plagiarism. It’s about teachers truly understanding and engaging with their students, fostering relationships that go beyond mere academics.

Would you encourage the use of AI?

Dr. Bell suggests that employing an artificial or virtual assistant could be quite beneficial. This tool could help verify information and provide feedback, allowing you to ask, “Here’s what I’ve written—does it effectively convey my message?”

In the past, individuals often felt lost and isolated, unsure of where to seek help. Today, however, there’s a helpful companion available to answer intermediate questions and provide valuable feedback. While some may overly depend on this resource instead of consulting friends or classmates—an extreme scenario—it can still be a useful tool when used appropriately. As long as we set boundaries, there’s nothing inherently wrong with leveraging this resource for support.

Prof. Robertson finds the situation quite thrilling. It presents a challenge: how can we enhance assessments to minimize the advantage of AI-assisted cheating? Instead of viewing AI as a threat, why not embrace it? By teaching students to effectively use AI, we can integrate its usage into assignments, acknowledging that it will likely be part of their work.

It’s time to shift our perspective on student learning expectations. With the rise of AI tools, we need to focus on how these technologies can enhance education rather than replace human effort. This means rethinking our assessment methods and redefining the skills and attributes we want graduates to possess. Embracing this change will help us prepare students for a future where collaboration with AI is essential.

How to check if something was written by AI

It’s evident that student essays showcase a richer vocabulary compared to those produced by AI, boasting over 35% more unique words than ChatGPT. But how can you tell if a piece of writing is generated by AI? Here are some key indicators to watch for.

Misinformation Alert: Have you ever come across a fact or statistic that seems off? It’s possible that AI has misinterpreted a false statement and incorporated it into the text. Students should be equipped with a solid grasp of their subject matter to identify these inaccuracies. Always take the time to verify any critical facts or contextual details to avoid including misleading information in your essay.

Boring language can be a telltale sign of computer-generated content. Our research shows that humans naturally use a much richer vocabulary, so if your writing feels repetitive or awkward, it may lack that human touch. To enhance your writing and break free from monotonous phrasing, consider taking conversational English lessons. They can help you elevate your flow and structure, making your work more engaging.

Redundant explanations: AI-generated writing can sometimes come across as overly verbose, filling space without adding real substance. This often manifests as lengthy explanations of concepts that lack solid facts or evidence, revealing the limitations of AI in delivering concise insights. However, human writers can also fall into this trap. To avoid sounding robotic, aim to provide rich details in each point while remaining clear and concise.

As AI content creation tools gain popularity, there’s been a corresponding rise in AI-detection technologies, including Turnitin, Copyleaks, and UniPapers. Although these sophisticated tools often keep their methods under wraps, they play a crucial role in assessing the extent of AI-generated text in student essays.

According to our experts, identifying AI-generated content can be quite challenging unless it exhibits clear signs of being robotic, repetitive, or factually incorrect. As the sophistication of AI writing tools improves, it becomes increasingly difficult for evaluators to discern whether a piece was crafted by a machine or a human, especially in longer texts, which often showcase significant variation in quality.

Through our analysis of the language patterns in human-written content and essays generated by ChatGPT, we have uncovered distinct differences in terminology and word choice. AI learns English from existing online sources such as web pages, blogs, and articles. Therefore, students should exercise caution when using AI tools, particularly for longer texts, to prevent the risks of inaccuracies and superficial content. Striking a balance between producing high-quality work and utilizing these tools wisely is essential to steer clear of plagiarism.

Elevate your vocabulary by exploring online language learning with LingualNeeds! Whether you’re eager to master conversational English, dive into traditional Spanish lessons, or gain confidence in French, we have you covered. Plus, our courses offer specialized options for learning business languages to enhance your professional communication skills.

Methodology

We gathered an impressive 12,346 student essays from IvyPanda, meticulously categorized by genre across 25 different subjects. This comprehensive collection allows for a detailed analysis of the data, which is organized as follows:

• Number of errors overall and per subject
• Most common errors overall and per subject
• Most common words used in essays overall and per subject
• Most common phrases used in essays overall and per subject

We only included essays that had a minimum of 600 words.

To compare the student-written essays with AI-generated essays, we tasked ChatGPT with composing 115 essays covering 25 subjects.

To identify errors in student-written essays, we employed LanguageTool, an advanced linguistic analysis tool. This allowed us to pinpoint the most frequent mistakes found in the essays, enhancing our understanding of common pitfalls in student writing.

Analysis is accurate as of April 2024.