A Genre, Scoring, and Authorship Analysis of AI-Generated and Human-Written Refusal Emails

Winny Wilson ; Heath Rose University of Oxford

Abstract

This study compares AI-generated (ChatGPT and Gemini) and human-written business refusal texts. A genre analysis found that AI-generated texts are formulaic and less nuanced than human-written texts. Applying a rating of professional writing quality, inferential statistics revealed no significant difference in scores between Gemini and human-written texts, but revealed ChatGPT as lower. Human assessors identified authorship of AI-generated texts with an accuracy rate of 68.1%, and human-written texts with 86% accuracy. Key concerns for assessors were tone, relationship, language choice, content, and structure. The findings inform four key areas of focus for teaching business writing in the AI age.

Journal
Business and Professional Communication Quarterly
Published
2025-03-12
DOI
10.1177/23294906251322890
CompPile
Search in CompPile ↗
Open Access
OA PDF Hybrid
Topics
Export

Citation Context

Cited by in this index (3)

  1. Business and Professional Communication Quarterly
  2. Business and Professional Communication Quarterly
  3. Business and Professional Communication Quarterly

References (45) · 7 in this index

  1. AlAfnan M. A. Dishari S. Jovic M. Lomidze K. (2023). ChatGPT as an educational tool: Opportunities challenges…
  2. Analysing genre: Language use in professional settings
  3. Bhullar P. S. Joshi M. Chugh R. (2024). ChatGPT in higher education—A synthesis of the literature and a futur…
  4. 10.1075/scl.28
  5. 10.1017/CBO9780511813085
Show all 45 →
  1. 10.1177/002194369002700403
  2. Campbell K. S. Naidoo J. S. Smith J. (2023). When your boss says “You need to sound more professional”: Writi…
  3. Business and Professional Communication Quarterly
  4. Business and Professional Communication Quarterly
  5. Casal J. E. Kessler M. (2023). Can linguists distinguish between ChatGPT/AI and human writing?: A study of re…
  6. Business and Professional Communication Quarterly
  7. Creelman V. (2012). The case for “living” models. Business and Professional Communication Quarterly 75(2) 176…
  8. de Rycker A. G. H. (2014). Mitigation in turning down business proposals across cultures: The case for pragma…
  9. Business and Professional Communication Quarterly
  10. Devitt A. J. (2015). Genre performances: John Swales’ genre analysis and rhetorical-linguistic genre studies.…
  11. Dhillon P. S. Molaei S. Li J. Golub M. Zheng S. Robert L. P. (2024). Shaping human-AI collaboration: Varied s…
  12. Flowerdew L. (2012). Exploiting a corpus of business letters from a phraseological functional perspective. Re…
  13. Fraser L. Harich K. Norby J. Brzovic K. Rizkallah T. Loewy D. (2005). Diagnostic and value-added assessment o…
  14. Business and Professional Communication Quarterly
  15. Gunser V. E. Gottschling S. Brucker B. Richter S. Gerjets P. (2021). Can users distinguish narrative texts wr…
  16. 10.1016/S0889-4906(99)00037-X
  17. Herbold S. Hautli-Janisz A. Heuer U. Kikteva Z. Trautsch A. (2023). A large-scale comparison of human-written…
  18. Hyland K. (2022). English for Specific Purposes: What is it and where is it taking us? ESP Today 10(2) 202–22…
  19. Introducing genre and English for Specific Purposes
  20. Journal of Business and Technical Communication
  21. Jansen F. Janssen D. (2013). Effects of directness in bad-news e-mails and voice mails. Journal of Business C…
  22. Jovic M. Mnasri S. (2024). Evaluating AI-generated emails: A comparative efficiency analysis. World Journal o…
  23. Knoth N. Tolzin A. Janson A. Leimeister J. M. (2024). AI literacy and its implications for prompt engineering…
  24. Köbis N. Mossink L. D. (2021). Artificial intelligence versus Maya Angelou: Experimental evidence that people…
  25. Are simple business request letters really simple? A comparison of Chinese and English bu…
    Text & Talk
  26. Nguyen H. Miller J. (2012). Exploring business request genres: Students’ rhetorical choices. Business Communi…
  27. Park S. Jeon J. Shim E. (2021). Exploring request emails in English for business purposes: A move analysis. E…
  28. Pinto Dos Santos V. B. M. (2002). Genre analysis of business letters of negotiation. English for Specific Pur…
  29. Rudolph J. Tan S. Tan S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher edu…
  30. Samraj B. (2014). Move structure. In Schneider K. P. Barron A. (Eds.) Pragmatics of discourse (pp. 385–406). …
  31. Journal of Business and Technical Communication
  32. Shelby A. N. Reinsch N. L. (1995). Positive emphasis and you-attitude: An empirical study. International Jour…
  33. Genre analysis: English in academic and research settings
  34. 10.1017/CBO9781139524827
  35. 10.1177/1461445609341006
  36. Vergaro C. (2004). Discourse strategies of Italian and English sales promotion letters. English for Specific …
  37. Evaluating AI and human authorship quality in academic writing through physics essays
    ArXiv Preprint ArXiv:2403.05458
  38. Ying J. (2020). The importance of the discussion method in the undergraduate business classroom. Humanistic M…
  39. Zhai X. Razali A. B. (2023). Triple method approach to development of a genre-based approach to teaching ESL/…
  40. Zhang Z. (2013). Business English students learning to write for international business: What do internationa…