Written Communication Oct 2024

Dense and Disconnected: Analyzing the Sedimented Style of ChatGPT-Generated Text at Scale

Ben Markey Carnegie Mellon University ; David West Brown Carnegie Mellon University ; Michael Laudenbach New Jersey Institute of Technology ; Alan Kohler

Abstract

ChatGPT and other LLMs are at the forefront of pedagogical considerations in classrooms across the academy. Many studies have spoken to the technology’s capacity to generate one-off texts in a variety of genres. This study complements those by inquiring into its capacity to generate compelling texts at scale. In this study, we quantitatively and qualitatively analyze a small corpus of generated texts in two genres and gauge it against novice and published academic writers along known dimensions of linguistic variation. Theoretically, we position and historicize ChatGPT as a writing technology and consider the ways in which generated text may not be congruent with established trajectories of writing development in higher education. Our study found that generated texts are more informationally dense than authored texts and often read as dialogically closed, “empty,” and “fluffy.” We close with a discussion of potentially explanatory linguistic features, as well as relevant pedagogical implications.

Journal
Written Communication
Published
2024-10-01
DOI
10.1177/07410883241263528
Open Access
Closed
Topics
online writing instruction artificial intelligence multilingual writers voice and style

Citation Context

Cited by in this index (1)

  1. Jiang et al. (2025)
    Does ChatGPT Write Like a Student? Engagement Markers in Argumentative Essays
    Written Communication

Cites in this index (11)

  1. Appleman (2023)
    A Brave New World Requires Courage: New Directions for Literacy Research and Teaching
    Research in the Teaching of English
  2. Eckstein et al. (2022)
    How Does the Language Control of L1 and L2 Writers Develop Over Time in First-Y…
    Written Communication
  3. Aull et al. (2020)
    From Cow Paths to Conversation
    Pedagogy
  4. Aull (2019)
    Linguistic Markers of Stance and Genre in Upper-Level Student Writing
    Written Communication
  5. Ching (2018)
    Tools Matter: Mediated Writing Activity in Alternative Digital Environments
    Written Communication
Show all 11 →
  1. Clayson (2018)
    Distributed Cognition and Embodiment in Text Planning: A Situated Study of Coll…
    Written Communication
  2. Sterponi et al. (2017)
    Endangered Literacies? Affordances of Paper-Based Literacy in Medical Practice …
    Written Communication
  3. Staples et al. (2016)
    Academic Writing Development at the University Level
    Written Communication
  4. Aull et al. (2014)
    Linguistic Markers of Stance in Early and Advanced Academic Writing
    Written Communication
  5. Selfe (1999)
    Technology and Literacy: A Story about the Perils of Not Paying Attention
    College Composition and Communication
  6. Haas (1989)
    How the Writing Medium Shapes the Writing Process: Effects of Word Processing o…
    Research in the Teaching of English
Also cites 32 works outside this index ↓
  1. 10.1057/9781137350466
     ↗
  2. 10.1016/j.jeap.2017.01.005
     ↗
  3. 10.37514/PER-B.2013.4791
     ↗
  4. 10.1145/3442188.3445922
     ↗
  5. 10.1017/CBO9780511621024
     ↗
  6. 10.1093/oso/9780195083644.003.0003
     ↗
  7. 10.1017/9781108686136
     ↗
  8. 10.1017/CBO9780511920776
     ↗
  9. 10.5054/tq.2011.244483
     ↗
  10. 10.1093/applin/amu059
     ↗
  11. 10.1016/j.rmal.2023.100068
     ↗
  12. 10.1017/CBO9781139107327
     ↗
  13. 10.1177/23294906231182616
     ↗
  14. 10.18653/v1/N19-1423
     ↗
  15. 10.3998/mpub.9708722
     ↗
  16. 10.1177/107769589304700403
     ↗
  17. 10.1016/S0889-4906(02)00017-0
     ↗
  18. 10.1016/j.jslw.2014.09.007
     ↗
  19. 10.1109/MIS.2009.36
     ↗
  20. 10.3366/cor.2013.0040
     ↗
  21. Holland D., Cole M. (1995). Between discourse and schema: Reformulating a cultural-historical approach to cul…
     ↗
  22. 10.1177/1461445605050365
     ↗
  23. 10.3998/mpub.9426902
     ↗
  24. 10.1109/TPC.2022.3228026
     ↗
  25. 10.1177/0731948715583115
     ↗
  26. 10.4324/9781315197692
     ↗
  27. 10.1162/tacl_a_00548
     ↗
  28. 10.1016/j.jeap.2013.12.001
     ↗
  29. 10.1080/09500789309541347
     ↗
  30. 10.18653/v1/N18-1202
     ↗
  31. Stanton C. (2023). A dis-facilitated call for more writing studies in the new AI landscape; or, Finding our p…
     ↗
  32. 10.37514/TWR-J.2023.1.1.02
     ↗
CrossRef global citation count: 15 View in citation network →

Related Articles

  1. Journal of Business and Technical Communication Jan 2026
    Increasing Literacy on the Scams Targeting Latines: Generative Artificial Intelligence, Digital Technologies, and the Latine Community ↗
    Gabriel Lorenzo Aguilar
    artificial intelligence technical communication multilingual writers voice and style
  2. Journal of Business and Technical Communication Jan 2025
    Rhetorics of Authenticity: Ethics, Ethos, and Artificial Intelligence ↗
    Adrianna Deptula; Paul Thompson Hunter; Richard Johnson-Sheehan
    multimodality artificial intelligence professional writing multilingual writers voice and style argument
  3. Philosophy & Rhetoric Sep 2024
    Vector Rhetoric: GPT’s Rhetorical Agency ↗
    Douglas Kulchar
    rhetorical theory quantitative research multimodality online writing instruction artificial intelligence multilingual writers
  4. Pedagogy Jan 2023
    Reimagining Classroom Participation in the Era of Disability Justice and COVID-19 ↗
    Clare Mullaney
    online writing instruction gender and writing disability studies multilingual writers voice and style
  5. Composition Studies 2023
    Truth-Telling: Critical Inquiries on LLMs and the Corpus Texts That Train Them ↗
    online writing instruction artificial intelligence multilingual writers