Written Communication Oct 2024

Dense and Disconnected: Analyzing the Sedimented Style of ChatGPT-Generated Text at Scale

Ben Markey Carnegie Mellon University ; David West Brown Carnegie Mellon University ; Michael Laudenbach New Jersey Institute of Technology ; Alan Kohler

Abstract

ChatGPT and other LLMs are at the forefront of pedagogical considerations in classrooms across the academy. Many studies have spoken to the technology’s capacity to generate one-off texts in a variety of genres. This study complements those by inquiring into its capacity to generate compelling texts at scale. In this study, we quantitatively and qualitatively analyze a small corpus of generated texts in two genres and gauge it against novice and published academic writers along known dimensions of linguistic variation. Theoretically, we position and historicize ChatGPT as a writing technology and consider the ways in which generated text may not be congruent with established trajectories of writing development in higher education. Our study found that generated texts are more informationally dense than authored texts and often read as dialogically closed, “empty,” and “fluffy.” We close with a discussion of potentially explanatory linguistic features, as well as relevant pedagogical implications.

Journal
Written Communication
Published
2024-10-01
DOI
10.1177/07410883241263528
CompPile
Search in CompPile ↗
Open Access
Closed
Topics
genre theory writing pedagogy artificial intelligence
Export
BibTeX RIS

Citation Context

Cited by in this index (2)

  1. Gallagher et al. (2026)
    Crafting and designing AI-simulated audiences in the writing classroom
    Computers and Composition
  2. Jiang et al. (2025)
    Does ChatGPT Write Like a Student? Engagement Markers in Argumentative Essays
    Written Communication

References (62) · 14 in this index

  1. Appleman (2023)
    A Brave New World Requires Courage: New Directions for Literacy Research and Teaching
    Research in the Teaching of English
  2. 10.1057/9781137350466
     ↗
  3. Aull (2019)
    Linguistic Markers of Stance and Genre in Upper-Level Student Writing
    Written Communication
  4. Aull L. (2020)
    How students write
  5. Aull et al. (2014)
    Linguistic Markers of Stance in Early and Advanced Academic Writing: A Corpus-b…
    Written Communication
Show all 62 →
  1. Aull et al. (2020)
    From Cow Paths to Conversation
    Pedagogy
  2. 10.1016/j.jeap.2017.01.005
     ↗
  3. 10.37514/PER-B.2013.4791
     ↗
  4. 10.1145/3442188.3445922
     ↗
  5. 10.1017/CBO9780511621024
     ↗
  6. 10.1093/oso/9780195083644.003.0003
     ↗
  7. 10.1017/9781108686136
     ↗
  8. 10.1017/CBO9780511920776
     ↗
  9. 10.5054/tq.2011.244483
     ↗
  10. 10.1093/applin/amu059
     ↗
  11. Clayson (2018)
    Distributed Cognition and Embodiment in Text Planning: A Situated Study of Coll…
    Written Communication
  12. Byrd A. (2023). Truth-telling: Critical inquiries on LLMs and the corpus texts that train them. Composition S…
  13. 10.1016/j.rmal.2023.100068
     ↗
  14. Ching (2018)
    Tools Matter: Mediated Writing Activity in Alternative Digital Environments
    Written Communication
  15. Clayson (2018)
    Distributed Cognition and Embodiment in Text Planning: A Situated Study of Coll…
    Written Communication
  16. Conrad D. (2001)
    Variation in English
  17. 10.1017/CBO9781139107327
     ↗
  18. DeJeu (2024)
    Topoi of Nonprofit Proposal Writing
    Business and Professional Communication Quarterly
  19. Devitt A. J. (2008)
    Writing genres
  20. 10.18653/v1/N19-1423
     ↗
  21. 10.3998/mpub.9708722
     ↗
  22. Eckstein et al. (2022)
    How Does the Language Control of L1 and L2 Writers Develop Over Time in First-Y…
    Written Communication
  23. Espinoza J. (2015, May 26). Erasers are an ‘instrument of the devil’ which should be banned, says academic. T…
  24. Firth J. R. (1957)
    Papers in linguistics, 1934-1951
  25. 10.1177/107769589304700403
     ↗
  26. 10.1016/S0889-4906(02)00017-0
     ↗
  27. 10.1016/j.jslw.2014.09.007
     ↗
  28. Gabrial B. (2007)
    Handbook of research on writing
  29. Gardner S. (2015)
    Corpus linguistics: Abstract book
  30. Gerrard L. (1989)
    Critical perspectives on computers and composition instruction
  31. Giltrow J. (2021)
    Academic writing
  32. Haas (1989)
    How the Writing Medium Shapes the Writing Process: Effects of Word Processing o…
    Research in the Teaching of English
  33. Haas C. (1995)
    Writing technology: Studies on the materiality of literacy
  34. 10.1109/MIS.2009.36
     ↗
  35. Halliday M. A. K. (1976)
    Cohesion in English
  36. 10.3366/cor.2013.0040
     ↗
  37. Holland D., Cole M. (1995). Between discourse and schema: Reformulating a cultural-historical approach to cul…
     ↗
  38. 10.1177/1461445605050365
     ↗
  39. 10.3998/mpub.9426902
     ↗
  40. Lang et al. (2023)
    Computer-Assisted Corpus Analysis: An Introduction to Concepts, Processes, and …
    IEEE Transactions on Professional Communication
  41. 10.1177/0731948715583115
     ↗
  42. 10.4324/9781315197692
     ↗
  43. 10.1162/tacl_a_00548
     ↗
  44. 10.1016/j.jeap.2013.12.001
     ↗
  45. 10.1080/09500789309541347
     ↗
  46. 10.18653/v1/N18-1202
     ↗
  47. Prior P. (1998)
    Writing/disciplinarity
  48. Radford A. (2018)
    Improving language understanding with unsupervised learning
  49. Russell D. R. (1991)
    Writing in the academic disciplines: A curricular history
  50. Selfe (1999)
    Technology and Literacy: A Story about the Perils of Not Paying Attention
    College Composition and Communication
  51. Stanton C. (2023). A dis-facilitated call for more writing studies in the new AI landscape; or, Finding our p…
     ↗
  52. Staples et al. (2016)
    Academic Writing Development at the University Level: Phrasal and Clausal Compl…
    Written Communication
  53. Sterponi et al. (2017)
    Endangered Literacies? Affordances of Paper-Based Literacy in Medical Practice …
    Written Communication
  54. S. V. G. (1908). Do your pupils use erasers? Western Teacher: Devoted to Schoolroom Methods. Practical Aids a…
  55. Vee A. (2023). Large language models write answers. Composition Studies, 51(1), 176–181. https://compositions…
  56. 10.37514/TWR-J.2023.1.1.02
     ↗
  57. Weaver W. (1949)
    Translation
CrossRef global citation count: 18 View in citation network → Build reading path →

Related Articles

  1. Journal of Academic Writing Dec 2025
    Student Evaluative Judgements of Writing and Artificial Intelligence: The Disconnect between Structural and Conceptual Knowledge ↗
    Christopher Eaton; Kaitlyn Harris; Erin Vearncombe
    genre theory writing pedagogy graduate education artificial intelligence
  2. Pedagogy Oct 2025
    Contributors ↗
    modern rhetorical theory rhetorical criticism genre theory cultural rhetorics first-year composition writing pedagogy advanced composition creative writing writing across the curriculum graduate education two-year college service learning teacher development technical communication professional writing labor and working conditions archival research multimodality artificial intelligence literacy studies race and writing gender and writing disability studies literary studies editorial matter
  3. Journal of Technical Writing and Communication Oct 2024
    Improving ChatGPT's Competency in Generating Effective Business Communication Messages: Integrating Rhetorical Genre Analysis into Prompting Techniques ↗
    Junhua Wang
    rhetorical criticism genre theory writing pedagogy professional writing artificial intelligence literacy studies
  4. Business and Professional Communication Quarterly Sep 2024
    AI Resume Writing: How Prompt Confidence Shapes Output and AI Literacies ↗
    Timothy M. Ponce
    genre theory writing pedagogy teacher development artificial intelligence literacy studies affect and writing
  5. Business and Professional Communication Quarterly Jun 2024
    A Rhetorical Approach for Reimagining Business Writing Instruction in the AI Age ↗
    Emily DeJeu
    rhetorical criticism genre theory writing pedagogy professional writing artificial intelligence