AI in writing¶
Delving into LLM-assisted writing in biomedical publications through excess vocabulary¶
{2406.07016} Delving into LLM-assisted writing in biomedical publications through excess vocabulary, Dmitry Kobak, Rita González-Márquez, Emőke-Ágnes Horvát, Jan Lause, Science Advances (2025)
Large language models (LLMs) like ChatGPT can generate and revise text with human-level performance. These models come with clear limitations: they can produce inaccurate information, reinforce existing biases, and be easily misused. Yet, many scientists use them for their scholarly writing. But how wide-spread is such LLM usage in the academic literature? To answer this question for the field of biomedical research, we present an unbiased, large-scale approach: we study vocabulary changes in over 15 million biomedical abstracts from 2010–2024 indexed by PubMed, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words. This excess word analysis suggests that at least 13.5% of 2024 abstracts were processed with LLMs. This lower bound differed across disciplines, countries, and journals, reaching 40% for some subcorpora. We show that LLMs have had an unprecedented impact on scientific writing in biomedical research, surpassing the effect of major world events such as the Covid pandemic.
Quotes
Our analysis is performed on the corpus level and cannot identify individual abstracts that may have been processed by an LLM. Still, the following examples from three real 2023 abstracts illustrate the LLM-style flowery language:
• By **meticulously** **delving** into the **intricate** web connecting […] and […], this **comprehensive** chapter takes a deep dive into their involvement as **significant** risk factors for […].
• A **comprehensive** grasp of the **intricate** **interplay** between […] and […] is **pivotal** for effective therapeutic strategies.
• Initially, we **delve** into the **intricacies** of […], **accentuating** its indispensability in cellular physiology, the enzymatic labyrinth governing its flux, and the **pivotal** […] mechanisms.
Notes
Here is my black list of excessive jargon (mostly ML-related) prior to the advent of LLM
opens up, paves the way, paves a new avenue, unveil, unravel, reveal, outperform, surpass, leverage, harness, showcase, unearth (!), meticulously, state-of-the-art, power, potent, intricate, unprecedented, innovative, hidden (depends on context), unique (depends on context), novel, insight, drive, control
Impact of ChatGPT on the writing style of condensed matter physicists¶
{2408.17325} Impact of ChatGPT on the writing style of condensed matter physicists, Shaojun Xu, Xiaohui Ye, Mengqi Zhang, Pei Wang (2024)
We apply a state-of-the-art difference-in-differences approach to estimate the impact of ChatGPT’s release on the writing style of condensed matter papers on arXiv. Our analysis reveals a statistically significant improvement in the English quality of abstracts written by non-native English speakers. Importantly, this improvement remains robust even after accounting for other potential factors, confirming that it can be attributed to the release of ChatGPT. This indicates widespread adoption of the tool. Following the release of ChatGPT, there is a significant increase in the use of unique words, while the frequency of rare words decreases. Across language families, the changes in writing style are significant for authors from the Latin and Ural-Altaic groups, but not for those from the Germanic or other Indo-European groups.
The results indicates that the release of ChatGPT does not lead to a statistically difference in writing quality between English and Germanic groups. [...] This suggests that Germanic speakers do not experience a significant improvement in their writing skills compared to English native speakers. A more probable reason could be that, since English is a sub-branch of the Germanic language and shares similarities with it, Germanic speakers may not feel a strong need to rely on ChatGPT. Similarly, in column (3), we observe that the speakers from other Indo-Euro language families do not show a significant improvement in their writing skills [...]. In contrast, for the Latin group (see column (2) of Tab. VI), we observe a significant improvement following the release of ChatGPT, compared to the English group. [...] A similar significant improvement is also noted for the Ural-Altaic group (see column (4) of Tab. VI).
Is ChatGPT Transforming Academics’ Writing Style?¶
{2404.08627} Is ChatGPT Transforming Academics’ Writing Style?, M. Geng, R. Trotta (2024)
Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT’s writing style in their abstracts through a statistical analysis of word frequency changes. Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis. The words used for estimation are not fixed but adaptive, including those with decreasing frequency. We find that large language models (LLMs), represented by ChatGPT, are having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of LLM-style abstracts is estimated to be approximately 35%, if we take the responses of GPT-3.5 to one simple prompt, “revise the following sentences”, as a baseline. We conclude with an analysis of both positive and negative aspects of the penetration of LLMs into academics’ writing style.
Quotes
How could the frequencies of words like “significant” grow significantly together? Another striking example is the frequency change of the words “are" and “is”. The counts in 10,000 abstracts of these two words were quite stable before 2023. However, the frequency of these two terms has dropped by more than 10% in 2023.
These examples, anecdotal as they are, may represent the tip of the iceberg of a wider and growing phenomenon: the rapid increase in the usage of ChatGPT or other LLMs. The rise and fall in frequency of specific technical nouns may well be related to the changing popularity of certain research topics, but that a research trend is responsible for the change in usage of adjectives appears implausible – even less so for words like “is” and “are”.