In the realm of artificial intelligence, the ability to detect whether a piece of writing has been generated using a large language model has been a challenging task for many AI companies. However, a recent breakthrough by a group of researchers has introduced a novel method for estimating the usage of LLMs in scientific writing. By analyzing the appearance of “excess words” that became more prevalent during the LLM era (specifically in 2023 and 2024), the researchers discovered that at least 10 percent of abstracts from 2024 were likely processed with LLMs.
To conduct their study, the researchers examined 14 million paper abstracts published on PubMed between 2010 and 2024. They tracked the relative frequency of each word across the years and compared the expected frequency based on pre-2023 trends to the actual frequency in abstracts from 2023 and 2024, when LLMs were widely used. The results revealed a significant increase in the usage of certain style words that had previously been uncommon in scientific abstracts. Words like “delves,” “showcasing,” and “underscores” saw a substantial surge in usage in 2024, indicating a shift in vocabulary patterns post-LLM introduction.
The researchers noted that while language evolution can naturally influence word usage trends, the sudden and extensive increases observed in post-LLM abstracts were unparalleled. In the pre-LLM era, such drastic year-over-year spikes were primarily associated with major world health events like the Ebola outbreak in 2015 and the Zika virus in 2017. However, the post-LLM period brought about hundreds of words with significant frequency increases that did not correlate to any specific global events.
Unlike previous studies that relied on external markers or human writing samples, this research leveraged a unique control group in the form of pre-2023 abstracts to analyze changes in vocabulary choice post-LLM. By identifying hundreds of “marker words” that experienced a surge in usage after the introduction of LLMs, researchers could pinpoint the distinctive characteristics of LLM-assisted writing. These marker words, predominantly verbs, adjectives, and adverbs, served as telltale signs of LLM incorporation in scientific writing.
Implications for Scientific Discourse
The findings of this study shed light on the transformative impact of large language models on scientific writing practices. With an estimated 10 percent or more of post-2022 papers in the PubMed corpus displaying signs of LLM assistance, it is evident that these advanced language models are reshaping the landscape of academic communication. The researchers emphasized that the actual percentage of LLM-assisted papers could potentially be higher, as some LLM-enabled abstracts may not contain the identified marker words.
The emergence of large language models has introduced notable shifts in scientific vocabulary and writing styles. By unveiling the distinctive linguistic changes accompanying LLM usage, this research opens up new avenues for understanding the influence of AI technologies on academic discourse. As we continue to navigate the evolving landscape of AI-driven writing tools, further investigations into the implications of LLMs on language and communication are warranted.
Leave a Reply